Original Post
I have googled this a bit and not been able to find much help, so I turn to the GD community ;) I have just added normals to my terrain renderer. The problem is, some of the calculations seem to be going wrong somewhere, but I can't figure out where. The effect I'm seeing is a sort of "checkerboard" effect (not quite a checkerboard though), parts of each quad seem to be lit completely differently from other parts, as if the normals for one or more of the vertices are way out of whack... heres a screenshot: 
The code for generating them seems pretty messy, so it wouldn't surprise me if something in it is wrong, I just can't seem to see it... here goes: Stuff before this is just converting image data to vertices (heightmap). indices is an std::vector<GLuint>, normals, facenorms, and verts are std::vector<double> This is all then put into a VBO and rendered in the usual way. If you need to see that code also please say so, but I don't think it's required. Any help on why some of my normals are borked would be appreciated :)
The code for generating them seems pretty messy, so it wouldn't surprise me if something in it is wrong, I just can't seem to see it... here goes: Stuff before this is just converting image data to vertices (heightmap). indices is an std::vector<GLuint>, normals, facenorms, and verts are std::vector<double>
//index the vertex data
for(int i = 0; i < m_ySize-1; i++)
{
for(int j = 0; j < m_xSize-1; j++)
{
indices.push_back(j+(i*m_xSize));
indices.push_back(j+1+(i*m_xSize));
indices.push_back(j+1+((i+1)*m_xSize));
indices.push_back(j+((i+1)*m_xSize));
//calculate the normals for this face
jVector t1, t2, t3;
jVector v1, v2;
rIter = indices.rbegin();
t3 = jVector(verts[(*rIter)], verts[(*rIter)+1], verts[(*rIter)+2]);
rIter += 2;
t2 = jVector(verts[(*rIter)], verts[(*rIter)+1], verts[(*rIter)+2]);
rIter++;
t1 = jVector(verts[(*rIter)], verts[(*rIter)+1], verts[(*rIter)+2]);
v1 = t2 - t1;
v2 = t3 - t1;
jVector norm = jMath::Cross(v1, v2);
norm.normalise();
facenorms.push_back(norm.x);
facenorms.push_back(norm.y);
facenorms.push_back(norm.z);
}
}
//now process facenorms and calculate vertex normals
for(int i = 0; i < m_ySize; i++)
{
for(int j = 0; j < m_xSize; j++)
{
jVector fn[4];
jVector vn;
//first the lower 2
unsigned int index = (j+(i*(m_xSize-1)))*3;
if(j<m_xSize-1 && i<m_ySize-1)
fn[0] = jVector(facenorms[index], facenorms[index+1], facenorms[index+2]);
if(i<m_ySize-1 && j>0 && index>=3)
fn[1] = jVector(facenorms[index-3], facenorms[index-2], facenorms[index-1]);
//now do the upper 2 making sure the index is not negative
int sindex = (j+((i-1)*(m_xSize-1)))*3;
if(sindex>=0)
{
index = (j+((i-1)*(m_xSize-1)))*3;
if(j<m_xSize-1 && i>0)
fn[2] = jVector(facenorms[index], facenorms[index+1], facenorms[index+2]);
if(i>0 && j>0 && index>=3)
fn[3] = jVector(facenorms[index-3], facenorms[index-2], facenorms[index-1]);
}
vn = fn[0] + fn[1] + fn[2] + fn[3];
vn.normalise();
normals.push_back(vn.x);
normals.push_back(vn.y);
normals.push_back(vn.z);
}
}