Original Post
I've decided to try to make a sincere effort to learn shader programming. I was using Cg to write a simple shader that did Blinn-Phong in Eye-Space for one light. It was easy in GLSL, but I've hit a snag with Cg. I had issues where certain regions of the camera transform could cause the lighting to "flicker". I narrowed the problem down to the transformation of my normals using another shader that just displays the normalized, eye-space normal vector as the fragment's color. Sure enough, the normal transformation was working properly for most camera transformations. (And by working properly, I mean that the normals appeared to face a sane direction and rotated as I moved the camera around them, as eye-space normals should). But for some camera transforms, the normals were being garbled. I'm a little weak on my linear algebra, but it almost feels as if my modelview matrix ends up being non-invertible for some transforms and was causing some sort of singularity or discontinuity. As far as I can tell, I am only translating, rotating, and _uniformly_ scaling my modelview matrix. Like I mentioned, I'm weak on the maths, but shouldn't this class of matrix always be invertible? Note the code below where I set the shader parameter for the inverse-transpose-modelview matrix. I was originally using the statement in the upper branch of the preprocessor if-statement. I was letting Cg grab the matrix and perform operations on it automagically. However, as a test, I decided to see what would happen if I inverted and transposed the matrix myself-- The is the code in the lower branch. To my relief, the modelview matrix is indeed invertible. This code block also does not suffer from discontinuities or strange behaviors. However, the resulting normal transformation does not appear to transform the normals into eye-space at all. And by that, I mean that the results are identical to displaying the normals as object-space normals with no transformation. Why might the CG command result in a different matrix? Am I correct that object-space normals need to be transformed by the inverse-transpose of the modelview matrix in order to be projected from object-space into eye-space?
#if ORIGINAL
// How I originally was filling out the normal matrix.
// This causes very strange behavior.
cgGLSetStateMatrixParameter(cgViewIT,
CG_GL_MODELVIEW_MATRIX,
CG_GL_MATRIX_INVERSE_TRANSPOSE);
#else
// Test case to see what happens when I handle the operation personally
// Results in very different behavior from the above.
{
mat4 View, ViewI, ViewIT, ViewITT, I;
I.identity();
glGetFloatv(GL_MODELVIEW, View);
ViewI = View;
ViewI.invert();
ASSERT(ViewI * View == I, "Not proper inverse");
ViewIT = ViewI;
ViewIT.transpose();
ViewITT = ViewIT;
ViewITT.transpose();
ASSERT(ViewITT == ViewI, "Not proper transpose");
cgGLSetMatrixParameterfc(cgViewIT, ViewIT);
}
#endif
Any idea what might be the problem here? Or at least thoughts on what I might want to look into next? I have a suspicion that the solution is either subtle or involves some fundamental misunderstanding. Thanks for taking the time to read this! P.S. I checked my matrix utility classes, and I'm pretty sure they're good. Not that I consider them to be above reproach...