Tutorial: Adding hack-free Stencil Shadows to Half-Life Last edited 8 months ago2024-04-02 11:07:11 UTC

So recently I had a eureka moment, and I realized I could add a stencil buffer into Half-Life without needing to use a hacked OpenGL dll, or an engine hook. The solution was rather simple: We have framebuffer objects available to us, which are off-screen renderbuffers that we can set a stencil buffer for ourselves, with the proper renderbuffer attachment. This means we can render to this framebuffer, and then copy it's contents into the main renderbuffer.

-TL;DR version:
In short, if you don't understand what I said above, we're working around Half-Life's framebuffer to create a stencil buffer, then copying the contents of our framebuffer into the one Half-Life uses for rendering. This allows us to have a stencil buffer.

Have some screenshots which show the effect in action: Before you begin:

-HL25:
In the third patch for this tutorial, I applied a code fix that automatically adjusts for the HL25 release changes that caused the crashes. Now you won't need a custom build for the 25th anniversary.

-Performance:
I did my best to optimize this implementation of stencil shadows, but since all vertex transforms and shadow volume extrusions are done on the CPU, performance might suffer with more detailed models, or with very complex scenes. Also you might not want all models to cast a shadow, so for props, it might be better to mark these with the Render Fx "No shadows".

-Compatibility with other effects:
This tutorial was designed with a clean SDK in mind, without any other special effects. The stencil shadow implementation depends on using framebuffer objects, which can conflict with other effects, like the Tron glow effect, or with certain code used by renderer replacements like Trinity. I cannot develop a workaround for every possible effect that modders use, so I'm afraid if you want to use both, you'll have to choose one or the other. If however you are a more advanced programmer and know how to work with FBOs and know your way around OpenGL and C++ enough, it is possible to fix these issues. I however, for obvious reasons, can't go into detail in this tutorial on how you can fix these, as it depends on the implementation and is out of scope for the tutorial.

I tried to make this tutorial as beginner-friendly as possible, so implementing it should not be difficult, despite complicated this all sounds.

This implementation has the following features:
- Stencil shadows can be cast both from light sources, and from the sun, the latter which it'll use if there are no light sources nearby.
- By placing your lights.rad file into your mod's folder, you can have the code generate a list of light sources automatically based on the entries in lights.rad and the geometry in the BSP file.
- This implementation adds the same MSAA anti-aliasing as you have in Steam Half-Life, except this will also work in WON.
- You can toggle the shadows with the r_shadows_stencil cvar.

If you want to test this effect before proceeding, please download the test mod in the vault entry below:

- For HL engine builds predating the 25th anniversary build: - For the 25th anniversary build: This tutorial will guide you through the steps necessary to implement this effect. First and foremost, head on to the following vault entry, and download the tutorial files: In the archive, under the "cl_dll" folder you will find a set of .cpp files. Extract these into your "src_dll/cl_dll" folder. Next, you'll find in the archive a folder named "common". You need to extract this to your "src_dll/common" folder. A file called "com_model.h" will be overwritten, but that is fine. Once done, add these files to your client project in Visual Studio, so they're compiled properly.

In order to have access to OpenGL functions on the linker side, we need to add the required library to the client dll. So in Visual Studio, go to your client project, then right click and go to "Properties->Linker->Input", and in "Additional Dependencies", after the other entries, add "opengl32.lib". You need to do this for both the Debug and Release builds.

So to start off with the code parts, we first need to add some code that manages light and light_spot entities for us, so that they're added to, and removed from the list of light sources properly. So, go to your "hl" project first.

There, open up cbase.h, and locate the following function definition:

virtual const char *TeamID( void ) { return ""; }

Right after that, add this:

// STENCIL SHADOWS BEGIN
virtual void    SendInitMessages(CBaseEntity* pPlayer = NULL) {};
// STENCIL SHADOWS END

Next, open up player.h, and locate this function declaration in the CBasePlayer class:

void CBasePlayer::TabulateAmmo( void );

Then add this new function declaration:

// STENCIL SHADOWS BEGIN
void InitializeEntities(void);
// STENCIL SHADOWS END

Then at the very end of the class after the definition for the m_flNextChatTime variable, add this:

// STENCIL SHADOWS BEGIN
BOOL m_sentInitMessages;
// STENCIL SHADOWS END

Open player.cpp. Here, look up the following line:

int gmsgStatusValue = 0;

Then add this:

// STENCIL SHADOWS BEGIN
int gmsgLightSource = 0;
// STENCIL SHADOWS END

Now go to the LinkUserMessages function, and at the very end of it, add the following:

// STENCIL SHADOWS BEGIN
gmsgLightSource = REG_USER_MSG("LightSource", -1);
// STENCIL SHADOWS END

What these do, is that they declare the client-side message, through which the server will inform the engine about any "light" and "light_spot entities, and whether they're switched on or off. This is needed, because we don't want invisible lights casting shadows.

We still need to take care of the code that ensures that players are updated with this information on-time. So locate the "CBasePlayer :: Precache" function, and at the very end add this:

// STENCIL SHADOWS BEGIN
m_sentInitMessages = FALSE;
// STENCIL SHADOWS END

This'll essentially tell the code that the player has spawned, and will need their data to be refreshed. This'll be done on a per-player basis whenever needed.

Locate the function, UpdateClientData, and at the very beginning, add this piece of code:

// STENCIL SHADOWS BEGIN
if (!m_sentInitMessages)
{
    InitializeEntities();
    m_sentInitMessages = TRUE;
}
// STENCIL SHADOWS END

This UpdateClientData function is called each frame, and the boolean we set will update the client about the "light" and "light_spot" entities, then mark the boolean as true, so we won't send any more data to the client.

Now just before the "CBasePlayer :: FBecomeProne" function, add this new function:

// STENCIL SHADOWS BEGIN
//=========================================================
// InitializeEntities
//=========================================================
void CBasePlayer :: InitializeEntities ( void )
{
    edict_t* pEdict = g_engfuncs.pfnPEntityOfEntIndex( 1 );
    CBaseEntity* pEntity;

    for(int i = 0; i < gpGlobals->maxEntities; i++, pEdict++)
    {
        if(pEdict->free)
            continue;

        pEntity = CBaseEntity::Instance( pEdict );
        if(!pEntity)
            break;

        pEntity->SendInitMessages(this);
    }
}
// STENCIL SHADOWS END

This function is what takes care of iterating through all the entities, and tells them to send their data for the current player.

The last file we need to edit on the server side is lights.cpp, so open it up. Locate this line:

virtual int        Save( CSave &save );

And above it, add:

// STENCIL SHADOWS BEGIN
void    SendInitMessages(CBaseEntity* pPlayer = NULL);
// STENCIL SHADOWS END

Then after:

int        m_iszPattern;

Add:

// STENCIL SHADOWS BEGIN
int        m_colorR;
int        m_colorG;
int        m_colorB;
int        m_brightness;
BOOL    m_isActive;
// STENCIL SHADOWS END

Just below, you'll find this:

DEFINE_FIELD( CLight, m_iszPattern, FIELD_STRING ),

Add these new lines here:

// STENCIL SHADOWS BEGIN
DEFINE_FIELD(CLight, m_colorR, FIELD_INTEGER),
DEFINE_FIELD(CLight, m_colorG, FIELD_INTEGER),
DEFINE_FIELD(CLight, m_colorB, FIELD_INTEGER),
DEFINE_FIELD(CLight, m_brightness, FIELD_INTEGER),
DEFINE_FIELD(CLight, m_isActive, FIELD_BOOLEAN ),
// STENCIL SHADOWS END

Locate the function "CLight :: KeyValue", and just before the "else", add this new if statement:

// STENCIL SHADOWS BEGIN
else if (FStrEq(pkvd->szKeyName, "_light"))
{
    int r, g, b, v, j;
    v = 0;

    j = sscanf(pkvd->szValue, "%d %d %d %d\n", &r, &g, &b, &v);
    if (j == 1)
        g = b = r;

    if (!v)
        v = 64;

    m_colorR = r;
    m_colorG = g;
    m_colorB = b;
    m_brightness = v;
}
// STENCIL SHADOWS END

Next locate the "CLight :: Spawn" and "CLight :: Use" function definitions, and replace those two functions with the new block:

// STENCIL SHADOWS BEGIN
void CLight :: Spawn( void )
{
    if (m_iStyle >= 32)
    {
        //        CHANGE_METHOD(ENT(pev), em_use, light_use);
        if (FBitSet(pev->spawnflags, SF_LIGHT_START_OFF))
        {
            LIGHT_STYLE(m_iStyle, "a");
            m_isActive = FALSE;
        }
        else
        {
            if (m_iszPattern)
                LIGHT_STYLE(m_iStyle, (char*)STRING(m_iszPattern));
            else
                LIGHT_STYLE(m_iStyle, "m");

            m_isActive = TRUE;
        }
    }
}

void CLight :: Use( CBaseEntity *pActivator, CBaseEntity *pCaller, USE_TYPE useType, float value )
{
    if (m_iStyle >= 32)
    {
        if (!ShouldToggle(useType, m_isActive))
            return;

        if (!m_isActive)
        {
            if (m_iszPattern)
                LIGHT_STYLE(m_iStyle, (char*)STRING(m_iszPattern));
            else
                LIGHT_STYLE(m_iStyle, "m");

            m_isActive = TRUE;
        }
        else
        {
            LIGHT_STYLE(m_iStyle, "a");
            m_isActive = FALSE;
        }

        SendInitMessages(NULL);
    }
}

extern int gmsgLightSource;
void CLight::SendInitMessages(CBaseEntity* pPlayer)
{
    if (pPlayer && !m_isActive)
        return;

    if (pPlayer)
        MESSAGE_BEGIN(MSG_ONE, gmsgLightSource, NULL, pPlayer->pev);
    else
        MESSAGE_BEGIN(MSG_ALL, gmsgLightSource, NULL);

    WRITE_SHORT(entindex());
    WRITE_BYTE(m_isActive ? 1 : 0);

    if (m_isActive)
    {
        WRITE_COORD(pev->origin.x);
        WRITE_COORD(pev->origin.y);
        WRITE_COORD(pev->origin.z);
        WRITE_BYTE(m_colorR);
        WRITE_BYTE(m_colorG);
        WRITE_BYTE(m_colorB);
        WRITE_COORD((float)m_brightness / 9);
    }
    MESSAGE_END();
}
// STENCIL SHADOWS END

Here we basically added code that ensures that "light" and "light_spot" entities do not get removed, even if they are not toggled lights. This is required, because the client otherwise has no awareness of any of these entities being in the level, so we need to do this for all of these entities.

That is it for the server side. Now you want to go into your cl_dll project, and open up hud.cpp. At the top of the file after the #includes, add these new includes:

// STENCIL SHADOWS BEGIN
#include "lightlist.h"
#include "svd_render.h"
#include "svdformat.h"
#include "svd_render.h"
// STENCIL SHADOWS END

Now locate the Hud :: Init function, and add these new calls after the MsgFunc_ResetHud call:

// STENCIL SHADOWS BEGIN
gLightList.Init();
// STENCIL SHADOWS END

This will take care of initializing any cvars and OpenGL functions that we need for our stencil shadows.

Scrolling down a little bit, you'll find the ~CHud destructor. In there, add:

// STENCIL SHADOWS BEGIN
SVD_Shutdown();
// STENCIL SHADOWS END

Next, in the VidInit function, add these after the call "GetClientVoiceMgr()->VidInit();":

// STENCIL SHADOWS BEGIN
gLightList.VidInit();
SVD_VidInit();
// STENCIL SHADOWS END

Next up, open up GameStudioModelRenderer.cpp, and at the top after the #includes, add:

// STENCIL SHADOWS BEGIN
#include "svd_render.h"
// STENCIL SHADOWS END

Then at the end of the R_StudioInit function after the call to g_StudioRenderer.Init(), add:

// STENCIL SHADOWS BEGIN
SVD_Init();
// STENCIL SHADOWS END

These function calls do essential cleanup functions, so that our OpenGL objects and our cache of shadow volume data objects are deleted on shutdown and on loading a new game. This is important so we don't have any memory leaks or memory left used in VRAM.

We're done with this file, so open tri.cpp, and at the very top after the #includes, add these new ones:

// STENCIL SHADOWS BEGIN
#include "svd_render.h"
#include "lightlist.h"
// STENCIL SHADOWS END

Next, go to HUD_DrawNormalTriangles, and add this line at the end of the function:

// STENCIL SHADOWS BEGIN
    gLightList.DrawNormal();
// STENCIL SHADOWS END

And in HUD_DrawTransparentTriangles, at the end add this:

// STENCIL SHADOWS BEGIN
    SVD_DrawTransparentTriangles();
// STENCIL SHADOWS END

These functions take care of calling the actual rendering parts. It would be very long-winded to go into details about what goes on here, but this is where we draw the in-shadow parts of the screen. The call to gLightList is solely for debugging purposes, which can be enabled with the "r_debug_lights" console variable.

Open up the view.cpp file, and at the top after the #includes, add these new ones:

// STENCIL SHADOWS BEGIN
#include "svd_render.h"
#include "lightlist.h"
// STENCIL SHADOWS END

Now locate the V_CalcRefdef function, and at the very end after the comment block with the SF_TEST #define, add these calls:

// STENCIL SHADOWS BEGIN
    gLightList.CalcRefDef();
    SVD_CalcRefDef(pparams);
// STENCIL SHADOWS END

The function V_CalcRefdef is called each frame before the world is rendered, allowing us the perfect time to do some setup functions, as well as to bind out stencil buffer framebuffer object, so that Half-Life will render into this instead of the main framebuffer.

Now comes the hardest part, where we'll need to modify the studiomodel renderer class. This'll be a bit more complicated, but I'll guide you through. Open up StudioModelRenderer.h, and at the top after the #endif, add these new lines:

// STENCIL SHADOWS BEGIN
#include <windows.h>

#include "elight.h"
#include "svdformat.h"
#include "gl/gl.h"
#include "gl/glext.h"

enum shadow_lightype_t
{
    SL_TYPE_LIGHTVECTOR = 0,
    SL_TYPE_POINTLIGHT
};
// STENCIL SHADOWS END

Next in the CStudioModelRenderer class definition, locate the definition for the function StudioProcessGait, and after that, add this new block:

// STENCIL SHADOWS BEGIN
    // Gets entity lights for a model
    virtual void StudioGetLightSources(void);

    // Sets bounding box
    virtual void StudioGetMinsMaxs(vec3_t& outMins, vec3_t& outMaxs);

    // Sets up bodypart pointers
    virtual void StudioSetupModelSVD(int bodypart);

    // Draws shadows for an entity
    virtual void StudioDrawShadow(void);

    // Draws a shadow volume
    virtual void StudioDrawShadowVolume(void);

    // Tells if we should draw a shadow for this ent
    virtual bool StudioShouldDrawShadow(void);

    // Sets up the shadow info
    virtual void StudioSetupShadows(void);
// STENCIL SHADOWS END

And at the very end of the class definition, add this block:

// STENCIL SHADOWS BEGIN
public:
    // Pointer to the shadow volume data
    svdheader_t* m_pSVDHeader;
    // Pointer to shadow volume submodel data
    svdsubmodel_t* m_pSVDSubModel;

    // Tells if a face is facing the light
    bool            m_trianglesFacingLight[MAXSTUDIOTRIANGLES];
    // Index array used for rendering
    GLushort        m_shadowVolumeIndexes[MAXSTUDIOTRIANGLES * 3];

    cvar_t* m_pSkylightDirX;
    cvar_t* m_pSkylightDirY;
    cvar_t* m_pSkylightDirZ;

    cvar_t* m_pSkylightColorR;
    cvar_t* m_pSkylightColorG;
    cvar_t* m_pSkylightColorB;

    // Array of lights
    elight_t* m_pEntityLights[MAX_MODEL_ENTITY_LIGHTS];
    unsigned int    m_iNumEntityLights;

    // Closest entity light
    int                m_iClosestLight;
    // Shadowing light's origin
    vec3_t            m_vShadowLightOrigin;
    // Shadowing light vector
    vec3_t            m_vShadowLightVector;
    // Type of shadow light source
    shadow_lightype_t m_shadowLightType;

    // Array of transformed vertexes
    vec3_t            m_vertexTransform[MAXSTUDIOVERTS * 2];

    // Toggles rendering of stencil shadows
    cvar_t*            m_pCvarDrawStencilShadows;
    // Extrusion length for stencil shadow volumes
    cvar_t*            m_pCvarShadowVolumeExtrudeDistance;
    // Tells if two sided stencil test is supported
    bool            m_bTwoSideSupported;

public:
    // Opengl functions
    PFNGLACTIVETEXTUREPROC            glActiveTexture;
    PFNGLCLIENTACTIVETEXTUREPROC    glClientActiveTexture;
    PFNGLACTIVESTENCILFACEEXTPROC    glActiveStencilFaceEXT;
// STENCIL SHADOWS END

Now comes the final part. Open up StudioModelRenderer.cpp, and at the top, add in the CStudioModelRenderer::Init function, add at the end:

// STENCIL SHADOWS BEGIN
m_pSkylightDirX            = IEngineStudio.GetCvar( "sv_skyvec_x" );
m_pSkylightDirY            = IEngineStudio.GetCvar( "sv_skyvec_y" );
m_pSkylightDirZ            = IEngineStudio.GetCvar( "sv_skyvec_z" );

m_pSkylightColorR            = IEngineStudio.GetCvar( "sv_skycolor_r" );
m_pSkylightColorG            = IEngineStudio.GetCvar( "sv_skycolor_g" );
m_pSkylightColorB            = IEngineStudio.GetCvar( "sv_skycolor_b" );

m_pCvarDrawStencilShadows = CVAR_CREATE("r_shadows_stencil", "1", FCVAR_ARCHIVE);
m_pCvarShadowVolumeExtrudeDistance = CVAR_CREATE("r_shadow_extrude_distance", "2048", FCVAR_ARCHIVE);
// STENCIL SHADOWS END

Then in the CStudioModelRenderer::CStudioModelRenderer constructor, at the end of the function, add this block:

// STENCIL SHADOWS BEGIN
m_pCvarDrawStencilShadows    = NULL;
m_pCvarShadowVolumeExtrudeDistance = NULL;
m_iClosestLight                = 0;
m_iNumEntityLights            = 0;
m_pSkylightColorR            = NULL;
m_pSkylightColorG            = NULL;
m_pSkylightColorB            = NULL;
m_pSkylightDirX                = NULL;
m_pSkylightDirY                = NULL;
m_pSkylightDirZ                = NULL;
m_pSVDSubModel                = NULL;
m_pSVDHeader                = NULL;
m_shadowLightType            = SL_TYPE_LIGHTVECTOR;

memset(m_pEntityLights, 0, sizeof(m_pEntityLights));

glActiveTexture = (PFNGLACTIVETEXTUREPROC)wglGetProcAddress("glActiveTexture");
glClientActiveTexture = (PFNGLCLIENTACTIVETEXTUREPROC)wglGetProcAddress("glClientActiveTexture");
glActiveStencilFaceEXT = (PFNGLACTIVESTENCILFACEEXTPROC)wglGetProcAddress("glActiveStencilFaceEXT");

if (glActiveStencilFaceEXT)
    m_bTwoSideSupported = true;
else
    m_bTwoSideSupported = false;
// STENCIL SHADOWS END

Next, locate the CStudioModelRenderer::StudioDrawModel function, and locate this piece of code:

if (flags & STUDIO_RENDER)
{
    lighting.plightvec = dir;
    IEngineStudio.StudioDynamicLight(m_pCurrentEntity, &lighting );

Right after this, add this new bit:

// STENCIL SHADOWS BEGIN
StudioGetLightSources();
// STENCIL SHADOWS END

You need to do the same for CStudioModelRenderer::StudioDrawPlayer, so locate this bit:

if (flags & STUDIO_RENDER)
{
    if (m_pCvarHiModels->value && m_pRenderModel != m_pCurrentEntity->model  )
    {
        // show highest resolution multiplayer model
        m_pCurrentEntity->curstate.body = 255;
    }

And right after that, add this:

// STENCIL SHADOWS BEGIN
StudioGetLightSources();
// STENCIL SHADOWS END

Next up, we need to take care of the weapon models players use, so find this line:

model_t *pweaponmodel = IEngineStudio.GetModelByIndex( pplayer->weaponmodel );

And add:

// STENCIL SHADOWS BEGIN
model_t* psavedrendermodel = m_pRenderModel;
m_pRenderModel = pweaponmodel;
// STENCIL SHADOWS END

Then after the following line:

*m_pCurrentEntity = saveent;

Add:

// STENCIL SHADOWS BEGIN
m_pRenderModel = psavedrendermodel;
// STENCIL SHADOWS END

Next up, locate the function CStudioModelRenderer::StudioRenderModel, and add in the very beginning:

// STENCIL SHADOWS BEGIN
StudioSetupShadows();
// STENCIL SHADOWS END

Finally, seek out the function CStudioModelRenderer::StudioRenderFinal_Hardware, and at the top of the function, add:

// STENCIL SHADOWS BEGIN
if( StudioShouldDrawShadow() )
{
    StudioDrawShadow();
}
// STENCIL SHADOWS END

Finally, we'll add a Render Fx flag which allows us to disable shadows on entities. Go into your "src_dll/common" folder, and in the file "const.h", look up the following line:

kRenderFxClampMinScale,        // Keep this sprite from getting very small (SPRITES only!)

And in the same enum, add this new Render Fx:

kRenderFxNoShadow = 101,

You'll want to add this new Render Fx to your FGD file as well, so open it up and locate this base class definition:

@BaseClass = RenderFxChoices

At the very end, to the existing list, add this new one:

101: "No shadows"

We're done with the coding part, so compile your DLLs and load them up in Half-Life. You should now have functioning stencil shadows that will work fine without any hacks needed. I hope you enjoy your new guilt-free shadows.

-Overfloater.