Original Post
(Note: I am using my own equivalent of the glu library for portability reasons; the function is pretty much exactly the same) I am confused what the purpose of the "WinZ" parameter to gluUnProject(WinXYZ,...,ObjXYZ&,...) is? As far as I know, the WinZ is supposed to be the distance I want the 'unprojected' world-space point (ObjXYZ) to be from the camera near plane, in "camera space" [is that the right terminology?] (0.0 = near, 1.0 = far). With the camera facing +Z from Z=0, with a frustum near-far of 0.1-5000.0, I pass a WinXYZ with WinZ as 1.0. The returned ObjXYZ point is around the far-plane at Z=~5000.0 (the result seems to be slightly inaccurate - I'll assume that's normal behavior for now). When I do WinZ=0.0, the returned ObjXYZ point is on the near-plane at Z=~0.1. Everything seems normal so far. But when I pass a WinXYZ with WinZ=0.5, the returned ObjXYZ remains on the near-plane at Z=~0.1, instead of what I wanted: Z=~2500.0. Anything other than WinZ=1.0 results in the ObjXYZ remaining on the near-plane. Q) Is this supposed to happen? Q) Is the slight inaccuracy in gluUnProject() normal? Thanks. My gluUnProject() equivalent, just in case there's an error I may have overlooked: The difference is that the inverse composed projection and modelview matrices are precomputed outside of the function
bool UnProjectVector(Vector3f& rkVec, Matrix4f& rkInvComposedMatrix, float afViewport[4])
{
Vector4f kIn;
Vector4f kOut;
kIn[0] = rkVec[0];
kIn[1] = rkVec[1];
kIn[2] = rkVec[2];
kIn[3] = 1.0f;
kIn[0] = (kIn[0] - afViewport[0]) / afViewport[2];
kIn[1] = (kIn[1] - afViewport[1]) / afViewport[3];
kIn[0] = kIn[0]*2.0f-1.0f;
kIn[1] = kIn[1]*2.0f-1.0f;
kIn[2] = kIn[2]*2.0f-1.0f;
kOut = rkInvComposedMatrix*kIn;
if(kOut[3]==0.0)
return false;
kOut[0] /= kOut[3];
kOut[1] /= kOut[3];
kOut[2] /= kOut[3];
rkVec = Vector3f(kOut[0],kOut[1],kOut[2]);
return true;
}