Original Post
Hey guys, I've (sorta) got picking to work. I found an implementation for gluUnProject: http://pyopengl.sourceforge.net/documentation/manual/gluUnProject.3G.html. I have been comparing my results with the results from gluUnProject, and my near plane projection is being calculated correctly, but my far plane projection isn't working. Here is my code:
float norm_x = ((2.f*fx/(float)m_iWidth) - 1.f);
float norm_y = ((2.f*(m_iHeight - fy)/(float)m_iHeight) - 1.f);
gsmatrix l = m_mProj*m_mView;
gsmatrix v = l.inverse ();
gsvector p1, p2, posn, posf;
p1[0] = norm_x*m_fNear;
p1[1] = norm_y*m_fNear;
p1[2] = -1.f;
p1[3] = 1.f;
p2[0] = norm_x*m_fFar;
p2[1] = norm_y*m_fFar; // <--- this isn't working
p2[2] = 1.f; // <--- this isn't working
p2[3] = 1.f;
posn = v*p1;
posf = v*p2;