Variance Shadow Maps

It's a way of representing a shadow map so that it can be linearly filtered - i.e. trilinear, aniso, mipmapping, or whatever other filtering you feel like (ex. gaussian blur). Thus by simply turning on trilinear and aniso filtering on that shadow map (on supporting hardware), one can get much improved quality at shallow angles or varying viewer distances.

Also by blurring the shadow map, one can get soft shadows at very little cost (separable convolution filter can be used => O(kernel size), and still a single texture lookup on read). Theoretically on ATI hardware one could also just turn on multisampled anti-aliasing when rendering the shadow map and get perfectly anti-aliased shadows (lack of driver support in OpenGL has been preventing me trying this...). Alternatively simply by using a LOD bias in the texture lookup, super-sampling can be used to smooth shadow edges.

Variance shadow maps are completely orthogonal to frustum-warping techniques such as perspective/trapazoidal shadow maps. In the paper we don't use any of these to keep the comparison fair, but I suspect that using a combination of these techniques would produce great results.

The one main issue that remains open is that of light bleeding, which can occur when there is significant depth complexity in light space (and is particularly noticable when a large filter kernel is used to blur the shadow map). In most scenes this is less objectionable than the ugly shadow map aliasing artifacts, but cases can certainly be constructed where it looks terrible :) There are a few work-arounds for now, but none of them nice. I think the ultimate solution will be to use the third and fourth moments (E(x^3) and E(x^4)) of the occluder depth distribution function (trivial) and to find a suitably tight inequality that uses these.

If you're going to be at I3D or GDC this year, I'd be happy to discuss the technique in person.

thank you for the explenation. Looks like a really good method. Hope I have time to implement it by GDC (always the stupid demo's for GDC which might prevent from it ;)), and if I have further questions I would really like to get in contact then.

PS: it seams like the paper went offline. Is there a reason or does anyone have a backup somewhere, which they can put online pls. (or you can directly send it to kenzo@larian.com)

Yeah it just moved to a more permanent home at:
http://www.punkuser.net/vsm/.

There's a stub at the previous address but the direct link to the paper won't work anymore :)

[Edited by - AndyTX on February 12, 2006 4:04:15 PM]

AndyTX,
sounds great.
So what you gain by using variance shadow maps is the ability to filter depth values.
If I render into a shadow map and the results of the depth comparison (== black and white values) into a screenspace RGBA render target, I should be able to bilinear filter this render target/texture with the hardware because it is a 32-bit texture. Wouldn't that be more efficient than bilinear filtering of depth values?
Let's say my target platform supports only 32-bit (8-bit for stencil) floating point depth buffers. So I can not use a 16:16:16:16 fp texture to store depth values. What I have to do now is generate two 32-bit depth buffer textures. Is that correct?
That would restrict your approach to NVIDIA graphics hardware and therefore excluding the PC market .... (the latest ATI graphics hardware does not support a 16-bit depth buffer format ... right?).

Did you get my e-mail?

- Wolf

Hi Wolf, this is William (the other author).

I would consider that a good concise summary of VSM - they allow you to linearly filter depth values.

Now if I can address your questions:

If you think about it, there's not much sense in rendering results of a depth comparison into a screen space buffer. Sure, you can filter the result - but all you will get are neighbouring _screen space_ pixels, instead of _light space_ texels. Perhaps it would be more efficient, but basically you are just blurring your scene, which will give totally wrong results.

Second, when we render the VSM we don't use a depth target. Depth targets are the wrong way to go because they support an odd set of formats, and may or may not have filtering, mipmapping, etc. It's a huge mess. Instead we just render the VSM values to floating point color targets. Therefore, the conclusion that we are excluding the whole PC market is somewhat premature. :)

If you have any more questions about the paper, let us know.

<<<
Therefore, the conclusion that we are excluding the whole PC market is somewhat premature. :)
<<<
upps I did not see this. Yes my conclusion was premature :-).

<<<
Depth targets are the wrong way to go because they support an odd set of formats, and may or may not have filtering, mipmapping, etc. It's a huge mess. <<<
I agree, but they save memory. If you render into a color texture, you need a depth buffer and a color texture in memory. Using a depth buffer only saves you some memory and speed (usually comes with a 2x draw speed) ... which might be crucial on some hardware platform.

Nevertheless your approach is very cool, because it opens up new horizons to optimize shadow map approaches on different platforms.

So I would love to get an article proposal for ShaderX5 from you :-).
- Wolf

Blurring the screen space result, post-depth-comparison is possible (see this article on GameDev.net for example) but unfortunately this approach will produce a "haloing" effect on shadow silhouettes (see the discussion of the article here) In addition it doesn't solve many of the shadow sampling issues correctly - i.e. instead of correctly applying mipmapping/trilinear/aniso filtering at the shadow map lookup step, the resulting incorrect projection is simply blurred to hide the sampling artifacts. Granted blurring the variance shadow map in light space will not produce "correct" penumbrae (with contact hardening), but it will produce larger blurs for larger regions in projected light space. It's also conceivable that correct penumbrae could be emulated cleaply with the LOD bias, but more research is needed on that front.

Regarding buffer formats, we do indeed use standard fp16 buffers at the moment, storing the actual fragment depth (in world space) and depth^2. fp32 buffers are also usable and avoid some numeric instabilities that can crop of with fp16 (but usually acceptable results can be achieved with fp16 using a few tricks).

As you have correctly noted, there is only native filtering support for fp16 textures on NVIDIA GeForce 6 and 7 series hardware. On ATI hardware we are forced to emulate bilinear filtering in the shader (which is pretty trivial, and quite fast). Mipmapping and anisotropic filtering can also be emulated, but that is more difficult and probably much slower. Still we expect future hardware to either support floating point filtering directly, or be fast enough when doing it explicitly in the shader.

Regarding using two 32-bit render targets (RGBA8 for example), I've been meaning to play around with that a bit. If the two numbers can be suitable encoded into these two rendering targets in such a manner that allows them to be linearly filtered and recovered with minimal loss in precision, this would be another way of doing VSM on hardware that doesn't support floating point texture filtering. I'd still recommend using standard "color" formats though rather than a "real depth buffer", especially since the latter often uses a non-linear fixed point format that may not be totally compatible with the math (but works fine if a simple comparison is all that is required).

Yes, we did get your e-mail, thanks :)

AP:

M_1 is just the "averaged value" of depth, and M_2 is just the "averaged value" of depth^2. The key is that, if you turn on filtering, the gpu will do the averaging for you.

So if you have a texture vsm with vsm.x = depth and vsm.y = depth^2, then all you need to do in the shader is:

// Read the moments from the texture
(m1, m2) = tex2d(vsm, shadow_texcoord)

// Compute variance by equation (4)
sigma_squared = m2 - m1*m1

// Compute contribution by equation (5)
shadow_result = max(d <= m1, sigma_squared / (sigma_squared + (m1 - d)^2)

Note that there's also an FX Composer file available on the VSM web site... it's very basic right now, but all the necessary shader code is there.

TODO in that file though (and if anyone does it, please send me the new one!):

1) Figure out a way to make FX Composer generate mipmaps and enable trilinear filtering.

2) Enable anisotropic filtering.

3) Potentially implement a separable gaussian blur as a demo of soft shadows using VSM.

The 3rd shouldn't be too bad, but I'm not familiar enough with FX Composer to know if the first two are even possible. Does anyone know?

euh .. 1 no idea, but when you set the filtering type in the sampler to anisotropic and you have configured your hardware to use anisotropic filtering (graphics card settings), then this should work no. Now you have:

sampler light_shadow_map = sampler_state
{
texture = ;
AddressU = CLAMP;
AddressV = CLAMP;
MipFilter = NONE;
MinFilter = LINEAR;
MagFilter = LINEAR;
};

You can chooce between the following, according to the DirectX SDK:

Quote:

D3DTEXF_NONE Mipmapping disabled. The rasterizer should use the magnification filter instead. D3DTEXF_POINT Point filtering used as a texture magnification or minification filter. The texel with coordinates nearest to the desired pixel value is used. The texture filter to be used between mipmap levels is nearest-point mipmap filtering. The rasterizer uses the color from the texel of the nearest mipmap texture. D3DTEXF_LINEAR Bilinear interpolation filtering used as a texture magnification or minification filter. A weighted average of a 2 × 2 area of texels surrounding the desired pixel is used. The texture filter to use between mipmap levels is trilinear mipmap interpolation. The rasterizer linearly interpolates pixel color, using the texels of the two nearest mipmap textures. D3DTEXF_ANISOTROPIC Anisotropic texture filtering used as a texture magnification or minification filter. Compensates for distortion caused by the difference in angle between the texture polygon and the plane of the screen. D3DTEXF_PYRAMIDALQUAD A 4-sample tent filter used as a texture magnification or minification filter. D3DTEXF_GAUSSIANQUAD A 4-sample Gaussian filter used as a texture magnification or minification filter. D3DTEXF_FORCE_DWORD Forces this enumeration to compile to 32 bits in size. Without this value, some compilers would allow this enumeration to compile to a size other than 32 bits. This value is not used.

hope this helps/works, else I have no idea how to set in FX composer.

Ok cool, thanks for the info... I'll try that out. Of course with the teapot model, aniso isn't going to make much difference (unlike the example given at the heading of the paper).

Hi everybody!

First of all, to the authors of the paper, it's a very interesting approach
to the problem with filtering/aliasing! Good work!!

Now to the questions....
I have just implemented the described technique (with a lot of help from the
.fx-file). Maybe not the most performance-effective version right now...,
but it works :) However, the technique seems to be VERY sensitive to high
depth-complexity... Any trix here, or is it just the way it is??

I will post some screenshots tomorrow and hopefully that will enlight my
problems. It's 5 AM here in Sweden right now, and I just finished the implementation, so the screenshot will have to wait for tomorrow...:)

/Mike

Quote:

Original post by vx7johmi However, the technique seems to be VERY sensitive to high depth-complexity... Any trix here, or is it just the way it is??

I'd have to see the specific image, but that's probably the "light bleeding" problem that we discuss in the paper. Basically with a bimodal (or more) distribution, Chebyshev's inequality is no longer a particularly tight approximation (since it uses only the first two moments of the distribution) and thus some light can "bleed" through where two occluders overlap in light space. This problem can range from small (for the sun => probably low depth complexity) to huge (for headlights => probably high depth complexity).

[Aside: light-space overlaps are actually a problem in many shadowing and related techniques.]

As for solutions, there are a few approaches. The first is to set up your scene so that it isn't a problem ;) This certainly isn't a *good* solution, but it can be done, and isn't altogether unlike stuff you have to do with other shadowing algorithms. Along these lines, the following will reduce the obviousness of the artifact:

1) Have some ambient or overlapping light in the shadow region (i.e. your lighting isn't going from 0 -> 1 for that single light's area). Anything that decreases the potential contrast of the light bleeding is "a good thing".

2) Don't make your light super-bright, and the only light in the scene. Same reasoning as above.

3) Don't move the light. To a lesser extent, don't move the objects in the scene, but this is usually unavoidable :) The light bleeding artifacts move relative to projected light space, so width a moving light source, it's much more obvious what's happening.

A second suggestion is to limit the size of your filter region. The size of the light bleeding can be only as large as the filter region in projected light space - i.e. if you're only using trilinear/aniso filtering, it'll be a maximum of one projected pixel wide. If you're using a huge gaussian blur kernel, the light bleeding can potentially be more extreme.

There are other potential approaches, like separating out different depth layers in the variance shadow map, but they kind of defeat the whole purpose of doing it that way. If hardware had programmable filtering (or if you want to do it yourself), there are some other approaches, like storing the max's/min's of the distribution, or other information that could help in obtaining a tight occlusion probability.

The ultimate solution is to find a tighter approximation that uses more moments of the distribution (which are trivial to obtain). We have potentially found a tight inequality for three moments, but the math is *significantly* more complicated. I'll post to the VSM web site (and probably this thread) if we come up with a 3 moment version.

So admittadly the light bleeding is a problem - probably the only significant one with the technique, which just makes it that much more annoying :) That said, the aliasing of other techniques is a fairly objectionable artifact as well (although people have gotten used to it somewhat), so which is worse will be somewhat scene-dependent.

Anyways I'll post back if we make any good progress on the 3 moment version - in the mean time definitely keep us informed of your results and experiences, and particularly methods to reduce or avoid the light bleeding problem.

Is it possible to implement this method on SM2.0 cards like a Radeon9800Pro oder X800? I am not quite sure because of the floating point texture filtering...

When can we expect a demo? :)
The fxcomposer workspace does not show anything for me (X800XT, Cat 6.2) :(

Quote:

Original post by Enrico Is it possible to implement this method on SM2.0 cards like a Radeon9800Pro oder X800? I am not quite sure because of the floating point texture filtering...

You naturally can't use hardware floating point texture filtering on cards that don't support it (i.e. all ATI and anything before GF6 series). However it's easy to emulate bilinear filtering in the shader, and pre-processing (blurring) the shadow map works everywhere. The only things that you'll lose are aniso and mipmapping... both of which *could* be emulated in the shader if required.

Quote:

Original post by Enrico When can we expect a demo? :)

Yes, soon :) Probably in the next month (I need to make one for I3D/GDC).

Quote:

Original post by Enrico The fxcomposer workspace does not show anything for me (X800XT, Cat 6.2) :(

Odd... what does it do? It works fine here, but looks the same as shadow mapping since the card doesn't support hardware floating point texture filtering :) I may update the fxcomposer file at some point to emulate bilinear in the shader, and/or do a gaussian blur. To be honest, I'm just familiar enough with fxcomposer to "make it good" :) It's basically just a source code demo.

Quote:

Original post by AndyTXYou naturally can't use hardware floating point texture filtering on cards that don't support it (i.e. all ATI and anything before GF6 series). However it's easy to emulate bilinear filtering in the shader, and pre-processing (blurring) the shadow map works everywhere. The only things that you'll lose are aniso and mipmapping... both of which *could* be emulated in the shader if required.

Ok, I have to rephrase my question: What is the advantage of VSM on hardware without floating point texture filtering?
I have just skimmed through your paper and it seems like this is a needed feature of graphics hardware to achieve good results?

Quote:

Original post by EnricoOk, I have to rephrase my question: What is the advantage of VSM on hardware without floating point texture filtering?

It still has the same advantages as without, but you have to do the filtering manually. The advantage of variance shadow maps IS the ability to linearly filter them, so you can either get the hardware to do it, or do it yourself :) So for example, using a gaussian blur to preprocess the shadow map and get soft shadows (with only one texture read per fragment) works just fine with hardware that doesn't support floating-point texture filtering.

Thus for said for such hardware, we implement bilinear filtering in the shader (trivial), and all pre-processing stuff works fine as well. The only thing that we *lose* over GF6/7 series cards is a bit of speed, and free mipmapping and anisotropic filtering.

Thanks again for the fast reply! :-)

Your fxcomposer workspace is working now. Don't know, what went wrong before :-/

Can you tell me please, what I need to change to get bilinear filtering for my card?

Quote:

Original post by Enrico Can you tell me please, what I need to change to get bilinear filtering for my card?

You just need to add shader code to take 4 texture samples instead of 1, and linearly interpolate based on the fractional part of the texture coordinate (in pixels). I'll update that soon-ish and maybe post some Sh code when I get home (that wouldn't be hard to turn into HLSL or whatever).