physics benchmark

Several years ago, someone (Chris Hecker and someone else I think, maybe Jeff Lander?) did a physics-engine face-off in Game Developer Magazine between Havok and MathEngine (then the two middleware physics options for games). This was more an evaluation of capability and robustness than a speed speed. I don't believe I've seen much beyond that.

I think it'd take quite a lot of work to do a fair benchmark, since the different physics engines all have different data structures, callback mechanisms, mixed support for collision detection, etc. The most reliable way to do a benchmark, to obtain fair results, would probably be something like the SPEC benchmarks for OpenGL, which test both raw theoretical performance and application-specific performance. Rather boring compared with the likes of 3DMark, but absolutely a more reliable comparison.

SPEC OpenGL Performance Characterization Project

I'm not aware of any effort to benchmark physics in a similar way, but wouldn't be surprised if someone is working on it.

A little while ago someone emailed me a program that did pretty much this - it contained various scenes that you could run using novodex, ode, tokamak, newton and jiglib, I think (may have got one or two wrong, but novodex and jiglib were definitely in there!). Unfortunately it wasn't complete - I think it must have been hard to get each engine to do the same things. Also, it's a hard comparison to make - all engines will give you some ability to trade off performance against simulation quality - which means to compare performance you have to compare quality.

I think the demo must be at home - I'll find it later and maybe contact the author to see if he has an update, or if he'd be prepared to open up hist test framework so others could add to it.

In theory the Rocket stuff is meant to be independant of physics API - it might be a good framework for someone wanting to do this.

Speaking about physics correctness I remember this thread

http://www.gamedev.net/community/forums/topic.asp?whichpage=3&pagesize=25&topic_id=291687

From few moth ago where the general consensus was that physics correctness was not possible because the law Vrel = -e * Vrel’ was empirical and therefore only impulsive methods can reproduce the correct behavior because the handle one contact/pair of colliding bodies at a time and therefore they can reproduce what the law say because the law only apply to two particles in isolation.

I happen to find this nice little Newton’ cradle demo

http://www.delphigl.de/files/newtoncradle.zip

boy it seems to agree with the real one in every time. I mean I have a real one here in my desk and they come real, real close.

So it seems the anonymous poster was not as wrong as it was shown in the thread, it looks like some engine do the physics better that others.

Different physics engines make different approximations for different reasons. They all "cheat" in the sense that they all approximate reality. There's nothing fundamentally different about Newton in this regard - maybe by default (with the original solver) it approximates less than other engines, but from demos I've seen you do pay in terms of performance. You can only judge the relative importance between accuracy and speed in the context of a particular game.

talking of newton, i managed to get rag dolls stuck in 2 seconds of whirling them around in one of their demos... also i noticed that the limps were moving out of their constraints (and back, of course) when whirling them around which i didn't manage using the rocket demo... any idea why this happens?

Agree just the term rigid body is already an approximation, so in that sense,since no collision is 100% elastic (rigid), no rigid body simulator can reproduce reality as it.

The point I was trying to make I that benchmarking an engine has to include more than just stacking few hundred boxes and see them in a frozen state.

It will be very interesting to see if with the new breed of hardware coming down the pipe, engines with more correct solvers will be able to overtake engines with iterative solvers.
A good case will be ODE which implement both an iterative and exact solver.

BTW: here are some videos I found on the same website of somewhat more advance elaborated simulations generated in a real time physics editor.

http://newton.delphigl.de/playground_gears.wmv
http://newton.delphigl.de/playground_planetarygears.wmv
http://newton.delphigl.de/playground_3catapults.wmv

It may be that the engine is cheating but if they are it is very well hidden, that collision system has me totally fooled

You can't do a fair comparison because the physics engines differ too much. In my tests, euler integrator is worse followed by mid-point which is 10x more accurate, followed by rk4 (not using adaptive stepsize) which is 23x more accurate than euler. I find rk4 to be ok. There is one or two better integrators that do adaptive stepsizes for accuracy vs. speed reasons but beyond that an advanced integrator would suck too much cpu power.

I was blown away by TrueAxis's demos, especially the car racing one which I find very addictive and fun, reminds me of f-zero game, and the ragdoll one where you get to toss the dummies around. I shot them straight up and watching them come down to be shot by another dummy. The newton racing car demo, the one with the blue jeep and circular race track I didn't liked at all. Those newton demos need lot more polish or less of them and more quality ones. Go and play the true axis's car demo because in that you can ram cars and go under them and simply have lots of fun. You can slip slide and the racing is very good too. Bumper to bumper in many cases. Btw, the ODE engine is ok but the tri-box collision is not very good for fast moving objects unlike true axis is. ODE needs better collision tri-mesh code, one that uses swept volume, point, etc. to prevent temporal effects.

Quote:

Original post by JDI was blown away by TrueAxis's demos, especially the car racing one which I find very addictive and fun, reminds me of f-zero game, and the ragdoll one where you get to toss the dummies around. I shot them straight up and watching them come down to be shot by another dummy. The newton racing car demo, the one with the blue jeep and circular race track I didn't liked at all. Those newton demos need lot more polish or less of them and more quality ones. Go and play the true axis's car demo because in that you can ram cars and go under them and simply have lots of fun. You can slip slide and the racing is very good too. Bumper to bumper in many cases. Btw, the ODE engine is ok but the tri-box collision is not very good for fast moving objects unlike true axis is. ODE needs better collision tri-mesh code, one that uses swept volume, point, etc. to prevent temporal effects.

i use newton in my project and i have to say this:

the true axis demos are truly amazing and quite accurate, but keep
in mind that true axis car demos use the raycast-car approach which
lets it model more "nicier" physics, probably all newton car demos
use a much more precise vehicle-joint, which provides much more
realism - it uses rigid bodies for tires ( while you prefer arcade fun )

newton car joint can as well provide fun and realism, stunt playground
demonstrates this, grab a copy of it here:
http://walaber.com/
newton dynamics forum topic for stunt playground:
http://www.physicsengine.com/forum/viewtopic.php?t=1143&start=0

As far as realism goes I find True Axis to be one of the less convincing SDK available. It seems to be using some enhanced Time Warp method where objects really slow down a few frames before collisions. It's not very stable and the interpenetration tolerance is a bit weird and let objects interpenetrate in bad looking configurations (eg: stack of cubes go to sleep in a 'curved' position as opposed to novodex greater tolerance but where two faces will always stick flat together).

The car demo from True Axis is however very good, far from perfect but still ways better than anything I tried based on Newton. Of course the physic library are not to blame here. Latest Meqon racing demo is not bad either but they have very bad troubles with polygon edges. Well at least True Axis and Meqon are playable, Stunt Playground is not imho.

Quote:

I do not think comparing how demos play is a good tool for engine comparison.

Wich is exactly what I meant by (and I quote) "Of course the physic library are not to blame here." as a matter of fact Buggy Boy was a very playable buggy (the vehicle) game and I'm sure there was no solid physic foundation to its gameplay.

Quote:

No engine can reproduce realistic vehicle physics, because tire model physics is not an exact science. However some engines do try to make a more realistic rigid body approximation than others.

Agreed, you may even refer to my posting in the thread about car physics wich still should be on the first page of this very same forum. And as a matter of fact I can't see (but you'll explain me he?) how having more realistic >rigid< bodies can help with modeling tire physic behavior.

Quote:

In fact the vehicle behavior in the Stunt playground is a lot closer to a real vehicle than in those demos.

Wich still doesn't make it any more playable.

Quote:

So as you can see comparing demos is more a comparison of the skill level of the person using the engine than comparing the engines. Especially when these demos are so different. The fact is that there is not standard for this yet.

Thanks, I can now see the light.

...

* and I promise to use an LCP solver * (sorry, had to)

Well it all depends on what the criterion of playable is. For some people a box sliding on a flat floor, that will not roll over even if it is hit at 120+ miles an hour by another “car” is a good vehicle simulation and a replacement for game play. I prefer the Stunt Playground.

I played the True Axis car demo and I do not think I need a physics engine to do that, I can tell you I do not think is possible to make the Stunt Playground without a good physics engine.

The way I see it with a more realistic rigid body simulator the tire model is more feasible because it is a lot easy to think of terms of degree of freedoms, tire torque, tire slip ratio, and things like that, which is absolutely impossible with a box on four springs.

Here is other case where Newton fidelity in the simulation is expected to match the reality. These are virtual simulation of the real stuff where the the Newton engine was chose amoung all the existing ones, and there are many more very serious institutions really checking it out.

http://robotics.ee.uwa.edu.au/auv/subsim/subsim/doc/SubSim.htm

Quote:

Original post by jovani Here is other case where Newton fidelity in the simulation is expected to match the reality. These are virtual simulation of the real stuff where the the Newton engine was chose amoung all the existing ones, and there are many more very serious institutions really checking it out. http://robotics.ee.uwa.edu.au/auv/subsim/subsim/doc/SubSim.htm

that is a 404 dead link.

newton is a good physics engine but it leaves a lot of things open
to you that you have to do it yourself, this is the only reason why
other physics engines may be better - newton is not for lamers :/

I'm guessing that most physics engines use a similar form of the equations of motion, with various tweaks to the friction model. They probably all use a Lagrange multiplier, constraint based formulation. This difference comes when you go to solve the equations of motion. Most engines use Gauss-Seidel, which is not accurate, but provides a smooth solution that adapts well to changing constraints. I wonder if anyone uses a Conjugate-Gradient algorithm. My experiments with it show that it does not adapt well to contact contraints.

Newton seems to have an accurate solver. Julio has said that it has O(n^2) cost so I'm guessing he does some sort of matrix factorization update each frame. It's too bad that he spends so much time proclaiming how good his engine is without telling us what his algorithm is and why it is better. I'm guessing it uses a quadratic progamming method, perhaps an interior point method.

Quote:

Original post by heeen has anyone tied all free available physics engines together to compare their performance in different situations? shouldn't be that hard..

Something like this?? (the screenshots don't show at the moment, the authors server seems to be down)

He's got 4 physics libraries running simultaneously: ODE, Novodex, True Axis and Newton.

So that we know what to avoid doing.

The problem with the jeep in Newton demo is that it doesn't have a reverse to get you out of situation that I happen to find myself in all the time. Driving in anything but rear drive mode is impossible because the jeep tends to oversteer and I can't drive a straight line. Going up hill takes forever. There is not enough space to accelerate and go thru those pipes. Newton has potential and it needs to tweak the jeep's physics to be game oriented not real world oriented as much. I was very frustrated by that demo and that is the last thing you want your customers to become imo.

Actually, I have implemented Featherstone's algorithms with loops for a commercial application. To do the loops and/or contacts you need to introduce Lagrange multipliers. So it's back to the same problem as the body coordinate formulation. Albeit with potentially far fewer variables.

It is not too difficult to deal with over constrained systems. Gauss-Seidel is not bothered by it. If you want the reactions to be well distributed, you can use an SVD or some other minimum norm approach. But this gets quite expensive, especially with inequality constraints.

I have studied Redon's article and I think it has a lot of promise. However, the solution methods I have found all boil down to a box-constrained QP problem, as in the Lagrange multiplier approach.

The demo of the two rings is nice. However, any engine with a GS solver and decent broad phase will handle that just fine. For a game, I would just use a ring of boxes.

Your last question gets right to the point. Why should any physics engine authors explain their algorithms? I think this is the sad truth of game physics. No one wants to share information. There are exceptions, such as Russ Smith (ODE) and Mr Rowl (Jiggle).

Quote:

Original post by JD So that we know what to avoid doing. The problem with the jeep in Newton demo is that it doesn't have a reverse to get you out of situation that I happen to find myself in all the time. Driving in anything but rear drive mode is impossible because the jeep tends to oversteer and I can't drive a straight line. Going up hill takes forever. There is not enough space to accelerate and go thru those pipes. Newton has potential and it needs to tweak the jeep's physics to be game oriented not real world oriented as much. I was very frustrated by that demo and that is the last thing you want your customers to become imo.

it is not newton's problem, it is stunt playground issuse, and to go to
reverse you have to use the gearbox.

The raycast car demos you speak of are simply that - demos. Of course you can make a better car using cylinders for wheels, fully hinged and spring laden suspension systems if you want. It seems quite naive to me to judge the engine's vehicle physics on simple demos when they're just one example of a car configuration - you don't HAVE TO use them like that. The guys who wrote True Axis wrote the physics for Carmageddon TDR2000 - they do know what they're doing and have shipped real world games.

Sure - and they do fantastic work too. All I'm saying (which applies to them as well) is don't judge the SDK on the demos alone, judge it on its capabilities in how you intend to use it. Granted, this is not easy but if it was, everyone would be doing it already :-)