Help with box stacking instability

You should try using 2 contact points, not just one. You'll also want to have some penetration tolerance instead of trying to achieve exactly zero penetration. You should also verify your own code against Box2D Lite (not recent Box2D, but the Lite version). This version only has OBBs and accomplishes what you are trying to achieve.

You can look at dBoxBox() in the ODE to find an example of a clipping algorithm. In a nutshell you find the axis of minimum penetration using e.g. SAT. I gave a presentation how to do this here: https://box2d.googlecode.com/files/DGregorius_GDC2013.zip. The axis of minimum penetration defines the reference face. Then you find the most anti-parallel face on the other shape. This defines the incident face. Finally you clip the incident face against the side planes of the reference face and keep all contact points below the reference face. You could also clip against the reference face, but those additional contact points don't add stability to the manifold from my experience.

Clipping can create more than four points, so you usually need to reduce the manifold. There is a good description how this can be achieved in GPG 4 by Pierre Terdiman and Adam Moravansky (both PhysX).

But as far as what I searched on the internet, it seems finding the deepest penetrating point is enough. maybe I miss something.

You can basically use incremental manifolds or full manifolds. Incremental manifolds were popular in earlier days, because they are easy to implement and relatively fast. Modern engines (e.g. also commercial ones) use full manifolds these days. This is not only for stacking stability. When you build an incremental manifold you need four frames to build a stable manifold. A manifold is stable if the mass centers project inside of it. You can see that even for a simple box on a plane you can get a manifold where the mass center projects onto an edge and the shape can rotate through the ground. This is the major issue with contact manifolds. From my experience objects don't fall out of the world because of high linear velocity as shown in text books. This is actually a trivial problem, but real world problems are much more subtle.

Thanks Dirk, that's really helpful.

do you know where I can find more information on this topic, especially discussion about stability? books or websites?

You can look at *all* presentations of Erin Catto and also at Box2D / Box2D Lite. You can find everything in the downloads section here: www.box2d.org

I gave GDC presentations about SAT and Quickhull which can be found there as well.

I have written a lot about these topics in public forums so searching for e.g. manifold, SAT, contact, clipping either here or in the Bullet forum might create some helpful results as well. The ODE mailing list archives might have also some useful stuff.

Finally, Randy might have some links to some relevant discussions and can share them here..

dBoxBox is a perfect place to figure out how to generate all the contact points. The code is hard to read so I wrote an article about it (see below). The trick for making contact points is to transform both OBBs into the model space of one of them. If you understand this all then you can greatly simplify the face clipping. This should all be in the ODE. Another place with an easier to read OBB to OBB manifold creation is in Ian Millington's Cyclone physics engine. There are like 4 or 5 different smaller easy to read functions that gather up the contact points. IIRC I don't think he did any clipping optimizations and does calculation in world space.

If you want stability then you'll just want to read Erin's stuff on solving constraints. Stability is largely about solving a system of equations, assuming you have proper collision detection. If you run sequential impulses on all the contact points you should have pretty good stability for starting out. You probably won't need to do manifold reduction immediately (reducing from a large number of contact points to only a few of the more preferable contact points), and can implement this later if needed.

For generating the contact points and doing collision detection for OBBs I wrote an article about this: http://www.randygaul.net/2014/05/22/deriving-obb-to-obb-intersection-sat/

But really if you don't understand the impulse solving and how to make it stable you should start with Box2D Lite and make sure you understand the contact caching, warm starting, and the contact constraint. Then, if you can write this code yourself and do the math, it will make sense when you try to do things in 3D. If you want to make sure you really understand the math you can take this naive solver here: https://github.com/RandyGaul/ImpulseEngine and convert it to use sequential impulses. It can be noted that the equations being solved are actually identical between the naive implementation and Erin's Box2D, but how they are solved (iteratively with solution caching, or naively) makes a big difference.

Dirk and Randy, thank you for the information, I think that's enough to get me started.