Hi, everybody.

I have a simple top down 2d game that have a bunch of characters and objects on screen with very simple shadows.
Shadows are basically inverted sprites tinted black and rendered at about 80 alpha.
It all looks ok until the shadows of multiple characters/objects start to overlap.. then it will create this effect:



How could I make the shadows not to stack on each other?

I'm using HGE (haaf's game engine) and one idea I got was to render all shadows to texture at solid black and then render this texture at half alpha between my level and characters.
This sounded initially like it could work but I hit the wall with HGE and render to texture that does not support alpha :/ or at least I had no idea how to render to texture with alpha working. I tried to play with blend modes, googled, but the best I could get is kind of white-ish shadow with BLEND_ALPHAADD which will get rid of all the black thus creates the desired "alpha" but leaves me with 'white' sprite that looks more like a reflection on water than shadow :/

any other ideas?
or any solution how to get that alpha work with render to texture in HGE?

I also googled and found a few posts on the original hge forum how to modify hge source to make alpha work but that failed too as I was unable to compile the hge (even without any changes..) as I got a bunch or errors..

One option would be to render all of your shadows onto an off screen gray-scale texture using a normal blending mode. You then draw that texture to the screen using a multiply with alpha letting you control how dark the shadow is. Unfortunately, the game engine you are using might not have the features to allow you to do that. I am not familiar with HGE so I cannot tell you if it would support that or not.

Use the stencil buffer to only render shadows where no shadow has yet been drawn. Set up the stencil buffer to clear to zero, and set up the stencil test to only render a shadow if the stencil value is zero and to increase the stencil value when a shadow is rendered.

A second shadow will not be rendered on top of an existing shadow, since the stencil value will not be zero for the overlapping areas. The final result is that the shadow effect for any pixel under any of the multiple shadows will be rendered exactly once.

Stenciling would have been my first inclination as well, but it sounds like you don't have that option within the confines of what you're working with..

http://hge.relishgames.com/doc/index.html?hgeconst_blendmode.html

It sounds like you already had it figured out, but just needed to use something more like BLEND_COLORMUL or BLEND_ALPHABLEND (making sure the shadow's pixels are black / dark gray).

If you already had a shadow working with additive blending, then it should be as simple as just using a different blend mode.

2_Robert Bob and radioteeth:

this is what I found about stencil buffer and hge: http://relishgames.com/forum/index.php?p=/discussion/3081/stencil-buffer-in-hge/p1

is there any other way how I could use it? any example how? I am quite new, still learning and it might be out of my skill...

2_HappyCoder: this sounds like something I have already tried to do.. I rendered my shadow sprite as opaque using 'render to texture'.. (lets say on with white background, since alpha is not working) then I slap this rendered texture on sprite and render it on screen with default blending and I got this:

now all I needed to do is to use like you said some kind of multiply blend mode that would get rid of the white colour and I could use alpha to get the desired transparency.

sadly, hge does not have any multiply blend mode (or I have no idea how to use them)..

I could only get rid of black from the pic above with using BLEND_ALPHAADD (or if I render my sprite at normal color on black background then the oposite effect):

I haven't been able to get the result I need with the hge blending modes available.

There is another idea that I got how to get rid of the white and thats colour-keying... of course hge does not support this, but I managed to find this code on their forum that should do it:

 void SetMaskColor(HGE *hge , HTEXTURE tex , int w , int h , int r , int g , int b)
{
int x , y;
    DWORD *color = hge->Texture_Lock(tex, false);
    DWORD maskcolor = ARGB(255, r , g , b);
    DWORD *pixel;
 
    for (x = 0; x < w; ++x)
    {
        for (y = 0; y < h ; ++y)
        {
            pixel = &color[y*w+x];
            if((*pixel) == maskcolor)
                (*pixel) = 0;
        }
    }
    hge->Texture_Unlock(tex);
}

good news is that this works! I have tried it and it works, BUT it only works with HTEXTURE which are textures.. and I need it to work with HTARGET which is render target.. my 'rendered to texture'

now I'm not very good at this but I try to do something like:

HTEXTURE newTexture;
...
newTexture = hge->Target_GetTexture(targetShadowMap); // I try to copy my HTARGET which contains my rendered texture into HTEXTURE

?

?

?

?

?

?

?

unfortunately when I pass this newTexture with the function above the whole thing will crash... on the line: if((*pixel) == maskcolor) with the message:

"Unhandled exception at 0x00d7c4f2 in Project2.exe: 0xC0000005: Access violation reading location 0x00000000."

If I use the function with any other HTEXTURE which I initialized as "hge->Texture_Load("texture.png");" then it works fine.