Skip to main content
GameDev.net gamedev.net
🔒 Locked

weird banding when using normalizing cube map

Started by muhkuh Feb 23, 2005 at 8:45 PM 12 replies 2.3k views
Original Post
muhkuh
muhkuh
Hi there, I'm scratching my head for two days now as I've got some problems using a normalizing cube map in my app. This is whatI do: -simple diffuse per pixel lighting in tangent space LxN -normal is fixed in tangent space to (0,0,1) for testing -L is renormalized by a cube map (resolution 128) -basis vectors a set up correctly I thing, I used meshmender, normals and uv coordinates are correct I wrote the shader in FXComposer where it worked as it should. The Problems start when I try to use it in my own app. There is some strange banding that moves when the model ratates (not in the same direction as the rotation). I noticed that this disappears when I do not renormalize L in the pixel shader with my cube map. The renormalizing cube map shouldn't be incorrectly created as I used the same in FX Composer without this ugly banding with the same shader. This is the pixel shader:

ps_1_3

def c0, 0,0,1,0
tex t0
tex t2
dp3_sat r0, c0, t2_bx2
This is the vertex shader:

struct vertexInput {
    float4 position			: POSITION;
    float3 normal			: NORMAL;
    float3 tangent 			: TANGENT;
    float3 binormal 			: BINORMAL;
};


struct VSOUT_sm11_pass0 {
    float4 hPosition		: POSITION;
    float3 light		: TEXCOORD2;
};

VSOUT_sm11_pass0 SM11_VS_pass0(	vertexInput IN, 
 				uniform float3 L) 
{
	VSOUT_sm11_pass0 OUT;
	OUT.hPosition = mul( float4(IN.position) , worldViewProj);
    
    float3x3 mObjToTangentSpace;
    mObjToTangentSpace[0]=-IN.binormal;
    mObjToTangentSpace[1]=-IN.tangent;  
    mObjToTangentSpace[2]=IN.normal;

    L=(mul( mObjToTangentSpace, L));
   
	OUT.light=L;

	return OUT;
}
When I didn't use the cube map I did 0.5*L+0.5 in the vertex shader and used "texcoord t2" instead of "tex t2" in the pixel shader. Any suggestions what could be the reason for the banding are welcome. Thanks, Markus
SimmerD
SimmerD
128x128 is a little small for the normalization cubemap. If you think about it, the maximum # of possible values a normal can take going left to right through an 8-bit texture is 256. That means that you want a 256x256 normalization cubemap. Also, make sure you are mip-mapping the cubemap as well, and trilinear may help. Make sure you aren't applying some sort of LOD bias to the cubemap, and that the texture is set to clamp.

This could be a limitation of 8-bit textures. Of course, with any sort of noisy diffuse texture this wouldn't show up very easily. The strange pattern could be caused by lod calculation, filtering, etc.

I am using cubemaps in my project with success, although I haven't tried perfectly white diffuse materials.

Using texcoord forces the vector to be clamped from floating point to fixed point per pixel, not losing any precision. Of course, it won't be normalized, so that's a tradeoff on this class of hardware. It's possible to pseudo-normalize the L vector after the texcoord instruction using math, but it's not clear that that wouldn't re-introduce some banding, although the pattern would be different.


JD
JD
This is normal. I think I run 512x512 cubemaps. The banding becomes more pronounced when using specular as well. Really, the 32x32 suggestion in Sim's paper :) I think, should be taken with a huge brick of salt ;)
muhkuh
muhkuh
Mmmm, I already have tried a 1024x1024 cube map as well with not much success.

What's the sense of mip mapping and trilinear filterning when using a renormalizing cube map? I just used a single mipmap level because I thought using a lower resolution cube map wouln't make sense with "function lookups" like this. I'll give it a try when I'm back home. I used this HLSL code for the texture.
samplerCUBE renormCubeMapSampler{    texture = <renormCubeMap>;    MinFilter = Linear;    MagFilter = Linear;    MipFilter = None;    AddressU = Clamp;    AddressV = Clamp;    AddressW = Clamp;}


Edit: Just read in an nVidia guide to normalization of vectors on GPUs that mipmapping is used for better performance with renormalizing cube maps. So I think not using mipmapping was the correct choice when trying to find problems with image quality.


I noticed the banding when I threw my shaders on a model. It is even VERY noticable when using a diffuse color texture. It just seems that the lighting is limited not to the 256 values of the pixel shader register resolution but to a much lower value so it changes quite roughly although the model rotates smoothly. Additionally some faces showed unstable lighting jumping up and down a bit although it actually should change smoothly or remain constant. Another thing that makes this very visible is that the banding moves. So I tried to find the reason for this and simplified my shader to this simple test case.

[Edited by - muhkuh on February 24, 2005 7:04:24 AM]
JD
JD
I wasn't able to remove the banding altogether either. It's just a less horrible looking when using higher maps then 32x32.
SimmerD
SimmerD
I should point out that mip-mapping is one of those cases where there isn't much of a speed/quality tradeoff of using it vs not using it. Using mip-mapping looks better and runs faster ( both due to reducing aliasing which causes visual sparkles as well as texture cache misses that hurt performance ).

Trilinear Mip-mapping can be slower than bilinear mip-mapping, but are higher quality, although modern cards have optimizations to reduce this hit as well.
JD
JD
Thumbs up for mipmaps :) Things in distance don't shimmer. Bilinear + mips are the starting point for me. Gl is pretty interesting in the filtering department, more so than d3d. Not sure if it's useful though :) The thing with bump maps is that filtering doesn't really help that much. You got to smooth them yourself in paint editor first for low res maps. Doom3 did really nice job with texturing and it's a really hard work to have it come out nicely.

Hey Sim, got a question for you. Do you guys put texture maps in local ram and how can I make sure. I just noticed in gl specs it said that they might show to be resident on first use but then might be demoted to agp ram. I'm worried that I have to count on agp/pcie bus as the speed limiter here. What happens with the texture priority slider? Zero marked textures go to agp and 1's marked go to video ram or what? I don't want to use gl's texture manager. Thanks.
SimmerD
SimmerD
I don't think we've ever done much with priority in the gl driver. I think we use it for choosing which ones to demote, and only support 0 and 1 right now.

The driver won't actively go around demoting textures just for fun, or just if you set the priority, rather only when you are out of local vid mem completely.

Nowadays on the low end, there is less and less difference between system and video ram, for instance, our TurboCache architecture on 6200 is a really nice fast way to access system ram over pci express, that acts as parallel bandwidth to the on-board video ram. There is no current way in d3d or ogl to tell if your textures are in local or pci-express memory.

On the high-end, there is more and more difference, as video memory bw outpaces system memory bw more every 6 months, so I understand that developers want to know what their local video memory footprint really is.

Scoob Droolins
Scoob Droolins
More questions about cube normalizers. I'm having alot of problems with these in SM1.0, horrible banding and blocky artifacts when used for specular.

1. How do you generate them? I've been using D3DXFillCubeTexture, and my callback does this:

D3DXVECTOR3 v;
D3DXVec3Normalize (&v, pTexCoord);
*pOut = D3DXVECTOR4 (v.x * 0.5f + 0.5f, v.y * 0.5f + 0.5f, v.z * 0.5f + 0.5f, 1.0f);

However, I cannot seem to exactly match an example normalizer map that I got from RenderMonkey.

2. The normalizer stage is CLAMP?
3. Any vector using the above normalizer must be biased (v * 0.5 + 0.5) before going into the pixel shader?
4. Vectors pulled from the normalizer are unbiased (_bx2) before use?

Finally, is there anything wrong with doing the specular calculation pow ((N dot H), spec power) like this:

// t2 : biased vertex normal
// t2 sampler : cube normalizer
// t3 : unbiased halfway vector
// t3 sampler : specular power table
tex t2 // vertex normal
texdp3tex t3, t2_bx2 // spec intensity

Thanks for any info you can offer.
JD
JD
Scoob, I've seen people use specular ramp texture to get rid of the banding. I think gamedev has some ati shader 1.1 articles on this but if not google this issue, you're bound to find something. Doom3 does it this way as well I think.

Sim, thanks for answers.
SimmerD
SimmerD
2. The normalizer stage is CLAMP?

Yes, true for any cubemap.

3. Any vector using the above normalizer must be biased (v * 0.5 + 0.5) before going into the pixel shader?

NO. You pass in regular signed texture coordinates, and the hw does the right thing. You only want to bias texcoords when passing into texcoord instruction or v0/v1 registers.

4. Vectors pulled from the normalizer are unbiased (_bx2) before use?

Yes, unless you make a signed cubemap.
muhkuh
muhkuh
Update:

As you said my cube map resolution was a little low. With 512 it's a lot better but still not perfect. While the banding has now a much higher frequency the problem with lighting not changing very smoothly remains. Unfortunately this cannot be seen on a screenshot. My testing model has some huge planar areas where this is quite visible. I will try using two 16bit cube maps to renormalize L as I've got a free texture stage anyway. May be I should skip renormalization completely in case of directional lights as it doesn't look that wrong without it.
muhkuh
muhkuh
Quote:
Original post by Scoob Droolins
More questions about cube normalizers. I'm having alot of problems with these in SM1.0, horrible banding and blocky artifacts when used for specular.

1. How do you generate them? I've been using D3DXFillCubeTexture, and my callback does this:

D3DXVECTOR3 v;
D3DXVec3Normalize (&v, pTexCoord);
*pOut = D3DXVECTOR4 (v.x * 0.5f + 0.5f, v.y * 0.5f + 0.5f, v.z * 0.5f + 0.5f, 1.0f);

However, I cannot seem to exactly match an example normalizer map that I got from RenderMonkey.

Looks alright to me. May be the rendermonkey one is signed and doesn't need the bx2 read modifier in the pixel shader? Just a guess.

Quote:

Finally, is there anything wrong with doing the specular calculation pow ((N dot H), spec power) like this:

// t2 : biased vertex normal
// t2 sampler : cube normalizer
// t3 : unbiased halfway vector
// t3 sampler : specular power table
tex t2 // vertex normal
texdp3tex t3, t2_bx2 // spec intensity

Thanks for any info you can offer.


Looks OK too. I did something similar. The only problem with this is that H is not renormalized per pixel causing unstable hightlights with very low tesselation.
Scoob Droolins
Scoob Droolins
Thanks for the help everyone. I tried making a signed cube normalizer, but it was still not the same as the RenderMonkey sample (strange, because an ATI engineer gave me that cube normalizer callback!) Also, I'm wondering if it makes any sense to go beyond a 256 pixel normalzier, due to the 8 bit res of each component?

Markus, like you, I am having the most serious artifacts on near-planar surfaces; well tesselated and more curved surfaces look pretty good. Despite the fact the the "tex t2; texdp3tex t3, t2_bx2" sequence does not allow H to be normalized, it is still quite useful for most cases. Plus, you wont find a two-instruction sequence that, in conjunction with the sampler contents, does more work :)

joe
image space

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.

Sign in to reply to this topic.