Hi,

I am trying to do ray casting with Ogre for the purpose of selecting a triangle on a mesh. I have this question.

As I understand it, ray casting is drawing a line from the mouse on the screen into the center of the model. The ray is tested first against a bounding box to find the mesh. However, I have 3 mesh objects, one inside the other and each would have it's own bounding box.

So how would I know which mesh to look at for where the ray intersects with which triangle?

Quote

As I understand it, ray casting is drawing a line from the mouse on the screen into the center of the model.

You cast a ray from the camera's eye position into the direction of the ray in the view frustum based on the mouse's location. (An easy way to do this is to project the point onto the near plane and using inverse World * View * Projection. Note not the World per object... only if you have global World matrix, otherwise just View * Projection) 

The ray is then intersected with bounding volumes to quickly cull a lot of geometry not in the bounding volume. If you have overlapping geometry, then their boxes overlap. You will then need to do a ray-triangle cast on the geometry in the bounding volume and determine the hit point you want (usually the closest hit point .e.g. origin + t * direction for minimum t (but not negative t))... but in any case the ray may intersect multiple triangles.

If you have a lot of triangles in your single bounding volume, it may be beneficial to create a bounding volume hierarchy if you're going to do a lot of ray casts (in order to speed it up.)

Alternatively if you don't really care about the triangle itself, but you care only about the object and you have trouble with the math... an easy 'solution' is to render a 1x1 viewport at the mouse location and render each object with a unique color. Then you can read back that color and you know which object was closest to the viewer. (The downside to this is that you need to use the GPU a little and you need to read back which causes a sync point. But in practice this is a fairly easy method for things like editors and the like.)

On 6/2/2019 at 10:12 PM, SR D said:

However, I have 3 mesh objects, one inside the other and each would have it's own bounding box.

You generaly omit only the uninterstected bounding volumes, and all intersected bounding volumes must be tested, you cannot even decide upon order of hit if they overlapp, only  unoverlapping volumes can be omitted if you are after the closest triangle.