color coded picking

I haven't explored every picking method out there, but if this were my project I'd approach it geometrically rather than mucking about with feedback mode or color-coding. With a simple octree implementation, you should be able to make the picking routine efficient enough for most purposes.

When you say highlight elements of the mesh, what constitutes an element? Do you only need to select triangles, or do you want to be able to select vertices also? What about edges? I just ask because this will affect (at least somewhat) whatever algorithm you choose to use.

I guess I'm just so unfamiliar with adding in an octree to any 3d model that gets loaded into my app, it seems like a really big job and I have no idea where to start. And I really don't know how that works from a user perspective, what steps the user goes through.

Wouldn't have any thoughts on that as well, would you?

By highlighting each element, I mean each individual triangle of the mesh as the mouse hovers over it. I also would like to be able to select vertices and edges as well.

You write code to generate the octree for the mesh automatically; the user doesn't have to do anything. Implementing an octree isn't entirely trivial, but it's not too hard (depending on your level of comfort with spatial subdivision schemes, I suppose).

Who knows, maybe you could get a color- or feedback-based method to work. It seems tricky though, especially since you want to be able to select vertices and edges. I suppose each of these would also have to be rendered with a unique color, and cover enough screen area to be 'pick-able'.

Again, I've never implemented feedback or color picking, so I probably can't help with that particular topic.

So, by selection I assume you meant something like this OpenGL FAQ entry?

Googling for color picking found this Lighthouse3D tutorial as well as this gpwiki entry.

Generally, both these methods render scene data to the back buffer with each object as a unique color, read the pixel data back for the mouse position, and search through the objects for that color. A key point being that you don't swap buffers so that the user doesn't have to see your unique colors rendering (and so you can rerender with your colors). Clear your color and depth buffers before rendering again.

A better method might be to make the color a (reversible) function of the triangle id in your triangle index list so that you can skip searching through your triangles for a color (which is probably fine for objects, as in those tutorials, but not for thousands of triangles).

To select an edge, assuming you don't have an edge list like you do a triangle list (and could use the color picking method on edges), you could first pick a triangle, and then compute the distance of the mouse position to the 3 edges of the triangle and pick the minimum.

To get your mouse position in 3-space you'll need the depth component from a call to glReadPixels(), and then call gluUnProject().

Next you'll want the distance to each edge (as seen here).



This should work regardless of which triangle you choose on either side of an edge. You can probably skip the sqrt in the distance computation, too, and just compare the squared distance.

To select a vertex you could probably use the same method on the 3 vertices of the picked triangle (but obviously using a different distance formula).

I'm not going to say either way whether you should use a geometric approach or a color picking method (both should be weighed in terms of complexity and your needs), but I hope this was of some help.