Intersection between Ray and Rectangle

What I would do is first perform the ray-plane intersection to get a point that you know is coplanar with the rectangle. Then you'd want to create a basis for the rectangle plane, convert the rectangle and the ray-plane point into "rectangle space", and then perform a standard 2D point-in-rectangle check. You could do this directly using standard transformation matrices, however you perform a bunch of extra calculations since this is essentially a 2D problem in 3D space, and with a little extra work you could reduce the math to a simpler form. I'm going to go with the matrix approach for now, since it's easier to explain and more "automatic" than expanding the math. If you want I could explain the other way as well, which takes advantage of the fact that we already have a point we know is in the rectangle plane.

Let's say that we have the rectangle points A,B,C,D in that order, and the ray-plane intersection point is P. Create a basis using the vectors [A-B], [C-B], and [A-B] x [C-B] (the cross product of the first two). Plug those three vectors into a 3x3 matrix in order and invert it. Take the inverted matrix and multiply it with (D - B) and (P - B), to create D' and P'. The reason we choose D is because it's the corner opposite the one with which we create the basis (B) and acts as a comparison point for later on, and the reason we subtract B is because the rectangle could be anywhere off in space, so we want to turn B into an origin for the rectangle space. Now we can do the standard 2D check - if P'.x >= 0 && P'.y >=0 && P'.x <= D'.x && P'.y <= D'.y, then the point is inside the rectangle. Otherwise it isn't. Hopefully I didn't miss anything [smile]