My rendering engine is coming along nicely, and now supports a simple scene graph with special light and camera nodes, and the light node supports any number of shadowing techniques. Calling render on the light node will do something like generate and render shadow volumes, render a depth map for shadow mapping, or six depth maps for omnidirectional shadow mapping using cube shadow mapping. Shadow mapping is now working perfectly with a single depth texture for spotlights, but my problem is this. After the six faces are rendered to the unrolled cube map (done with six 2D textures), I am having a lot of trouble with face selection.
I understand how it should work theoretically, but for some reason just can't get it working. But instead of asking you to fix my code, I'd just like to ask for a snippet or two of GLSL code from someone who's been there and done that. So assuming that my cube can be rotated arbitrarily, and I can get the vector from the object to the light (per vertex as a varying), about how should the code for face selection look in GLSL? I am something of a newcomer to GLSL.
Another small question too, would it be okay to just pass the view matrix of the camera for the rotation of my shadow cube, and extract the 3x3 rotation matrix in the shader? How would this be done in GLSL?
Sorry for being such a GLSL newbie, but anything concrete would really help me wrap my head around this problem.
Thanks in advance.