The better part of the day I spent trying to get mesh-mesh collisions going in Ageia PhysX; between our level and some arbitrary object. The thing just went mad with jittering, bouncing, and seemingly random behaviour.
I assumed it was something to do with the way the engine sets up the physics system and objects (skin width, center of mass, inertia tensors, something like that). No luck tweaking values or setting flags.
It all boils down to this. In PhysX, for mesh mesh collisions, you need to create pmaps for the meshes. pmaps only work properly on meshes that represent some sort of closed volume. As far as I can understand from the documentation, the process of generation applies raycasts to find extents of the volume, and further stores this in some sort of voxel structure for faster and more robust intersection tests. Now, our level naturally isn't a closed volume mesh, so ultimatly that screws up the pmap and any hope of arbitrary meshes colliding with the world [smile]
To resolve this I suppose we could remodell the level with closed volume colission meshes, which in turn might make the character interaction less smooth (pluss our level designer will probably be jumping with joy [rolleyes]). Perhaps more realistically, we can restrict all world-interacting bodies to be of convex shapes, which should work fine without pmaps.
None of this is too hard in itself, but working through someone else's wrapper (the engine we're using), hacking and slashing through their indirections, adding here and there, and never quite knowing if a problem stems from a bug in our code, their code, or in Ageia's PhysX itself, makes for a somewhat frustrating experience.
May tomorrow be a better day [rolleyes]
I wish you better luck for tomorrow. :)