Original Post
I have a problem.. I've been working on trying to get my picking algorithm to work for an age now with no luck.. I'm trying to figure out the math necessary to create a picking ray (e.g. rayStart, rayEnd) from 2D screen coordinates. I've tried a few methods so far & every single one has pretty much failed to generate reasonable values.. Here's the current state of my code..:- The function seems to generate values of rayStart = { -0.0f; -0.05f; -0.0f } rayEnd = { -0.0f; -0.05f; -0.0f } With a camera position of { 0.0f; 10.0f; 20.0f; } facing the origin.. These values seems incredibly low considering rayStart should be a point on the nearplane (at 0.1f) & rayEnd on the farplane (at 5000.0f) respectively.. I'm not quite sure where it's going wrong here.. Any help you could provide would be very much appreciated.. Cheers
void ScreenToWorldRay( Point2& screenpos, Vector3* rayStart, Vector3* rayEnd )
{
F32 fNear = 0.1f;
F32 fFar = 5000.0f;
F32 fWidth = GetCanvas()->GetWidth();
F32 fHeight = GetCanvas()->GetHeight();
Vector3 pos;
pos.setX( (screenpos.X/fWidth)*2 - 1.0f );
pos.setY( 1.0f - (screenpos.Y/(fHeight/2)) );
pos.setZ( 0.0f );
Vector3 camPos = GetCameraPosition();
Matrix4 mw = Matrix4::Translation( camPos );
Matrix4 mv = GetTransformMatrix( Matrix_View );
Matrix4 mp = GetTransformMatrix( Matrix_Projection );
Matrix4 inverse = mw * mv;
inverse = Matrix4::Inverse( inverse * mp );
Vector4 rs = (inverse * Vector3(pos.X * fNear, pos.Y * fNear, fNear));
Vector4 re = (inverse * Vector3(pos.X * fFar, pos.Y * fFar, fFar));
*rayStart = (rs.getXYZ() / rs.W);
*rayEnd = (re.getXYZ() / re.W);
}