Right Angle of a plane/rectangle

Ax+By+Cz=D is the equation of a plane. What you're calling right angles is commonly referred to as normals. (A,B,C) is the normal for the plane. D*(A,B,C) is the closest point on the plane to the origin. If A^2+B^2+C^2=1 then (A,B,C) is a unit normal. A point (X,Y,Z) lies in the plane if AX+BY+CY=D. If it's less than D it lies on one side, if it's greater than D it lies on the other. If D is always positive the less is always closer to the origin than the plane and greater is further away. Three points define a plane, i.e. V1, V2, V3. N=(V2-V1)x(V3-V1) where x is the cross, or vector, product of the two vectors, is a normal for that plane. It may point towards or away from the origin though. If N.V1 where . is the dot, or scalar, product of two vectors is negative it points towards the origin and you should use -N instead. If it's zero it's arbitrary, but it helps be consistant so one rule is it can only lie in four of the eight octants defined by the coordinate planes or is one of the positive coordinate axes. The four octants are the ones sharing edges with two positive coordinate axes. That way there is no uncertainity as to what the sign of the dot product means.


So there you have an equation of a plane defined by it normal and how to find it. Your rectangle is a region within the plane. That can be viewed the same though. Even though the edge is a line the normal gives you a plane. Rather than direction and distance from the origin it's direction and distance from the closests place on the plane to the origin. That allows you to create coordinates within the plane. If it truly is a rectangle then that rectangle is axes aligned within that coordinate plane. So it's just a<=X<=b and c<=Y<=d to see if the point lines in or on the edge of the rectangle. Rectangles are a bit restrictive, but if you just view an edge combined with the normal defines a plane it's all basically the same except the bounds check is a little more complex. You have to check it's on the right side of all the planes bounding the region or lies within one or more planes, i.e. a vertex is in two planes. The restriction then is simply a convex polygon. If it's concave, i.e. there's indentations, then you can be on the wrong side of some planes and still in the polygon. Easiest there is break a concave polygon into multiple convex ones.

Ray tracing a mesh is a bit silly though. Dot products and cross products in three space can do a whole lot as long as it's a flat surface. The advantage of ray tracing is infinite detail for curved surfaces. With curved surfaces you need calculus, specifically derivatives. A fair simple application can expand significantly what you can do, i.e. tori, ellipoids, quadradic surfaces. It's basically the same as above. To do computer graphics of any reasonable complexity you have to use linear algebra and derivatives is no more complex. This plenty to learn though without getting into complex surfaces. Even so eventually you hit a point where the only "more" to do is more complex scenes. When you get there if seems impossible remember there's calculus.