Original Post
I am working on 3D mouse selection for my current engine and I have decided to use a ray from the viewer through the mouse coordinates to determine a hit detection vector. I think the method I'm implementing should work, but I'm going to throw it out there for anyone to correct me if I'm wrong. I have a viewer which is viewing a scene. The viewing vector originates from the center of the viewing space. I use the X,Y mouse coordinates to determine an angle from the current viewing vector. For example, I have a standard viewing angle/frustum (45 degrees), as I move the mouse around the screen, I use the X,Y mouse coordinates to determine the deviation angle (to a max of 45 degrees in any direction) from the center of the viewing space. So, for example, I am looking down 30 degrees at a scene, I move the mouse to the upper left corner of the screen (0,0 window coordinates), my mouse vector would be the current viewing angle of 30 degrees down (or -30 degrees) and 0 horizontal plus my deviation of 45 degrees vertically and -45 degrees horizontally which would be 15 degrees up and -45 degrees (or 315 degrees if keeping positive units) horizontally. Thus gives me the angle of the vector to use to caculate a hit vector for mouse selection. I'm still trying to get my head around this one, so if there are holes in my master plan, please shine a light at them. Thanks.