Benchmarks

What the!? 9ms to call a funcion in AS (from C/C++ that is)? Am I the only one who thinks that sounds kinda much? Can you please post that benchmarking code? I have to see if the same thing happends with later versions as well..

No, that's 9 microseconds.

1000 microseconds = 1 millisecond. Each call took 9 microseconds, or 0.009 milliseconds.

And, no, it's not a lot. The overhead of asIScriptContext::prepare is well known. It's doing a lot of work. And, as you can see, AS trumped it in most of the other tests - most noticably the calls back into C++.

Yes, sorry, my bad.. I was absolutely sure it was measured in milliseconds.. While I do understand it takes time to prepare and things, 9ms seemed extreme.. 9µs however isn't very much at all.. Again, thanks for pointing this out, I'll read more closely before I make a statement next time (rating-- on self :p)..

Here is the whole project: vc7(2003).

Ia a bit messy, (...) math float tests have room of improvement.

I was interested in call times between c->script
script->script and script->c
Project can be imported in code::blocks NP.

I zipped the whole structure dir with libs and
headers. The exe is gen in /_bin
All libs are generated in /_lib
If you step in debuger make sure you set the exeutable path
to ../_bin couse the dumb vc7 holds the absolute path.

BM_.rar

[Edited by - Marius on May 26, 2006 3:10:26 PM]

Quote:

Original post by Anonymous Poster Pretty much a year ago, we had this thread: http://www.gamedev.net/community/forums/topic.asp?topic_id=311655&forum_id=41&gforum_id=0

Didn't see any benchmarks there...

Cool! [grin]

This really shows me where I have to spend some time optimizing the code for AngelScript. :)

I'll also take a look at that benchmark code you have, to see if you're doing everything the optimal way. ;)

Regards,
Andreas

Interresting comparison, I would wish someone would also test a couple of other scripting languages which "compete" which AS.

I don't know much about the techniques in either AS or Squirrel, but one interresting thing to note is that the void function takes much longer in AS, so one must assume that AS does quite a lot more than Squirrel, which makes the other tests unfair. For instance AS might generate some internal byte-code, while Squirrel just store the functions in a string and doesn't parse it until it's actually run. Of course it could also be a possibility that AS just parse the function everytime it's called and Squirrel actually notes that the function have been parsed once, I don't think this is what is happening though since "Script calls another script void function 100000 times" also should be slow in AS then.

Quote:

Original post by Marius microsft c++ vc7 compiler 2003 edition,

Might be a little off-topic, but the 2003 edition is called VC 7.1, VC 7.0 is the 2002 edition, which one do you use?

For simple benchmarks, it is known that Lua can be faster than Squirrel. Here are Alberto's remarks regarding benchmarks and real-world applications (such as large MMO with several megabyte of data living in the VM, etc.):

From:
http://www.squirrel-lang.org/forums/827/ShowPost.aspx

Quote:

As a personal experience, I'm writing an MMOG, and we also have a huge amount of squirrel code. As a matter of fact our game is written is squirrel and there isn't any part of the C++ that is specifically tight to our current title. As you can imagine we have a lot of entities (how we call game objects with a script attached) plus main loop, DB interface etc... So far squirrel doesn't really show up in the profiler. As any game what is consuming all our CPU is Graphics on the Client and Collision detection and Network protocol on the server. Now talking about squirrel performances, I'm personally very satisfied at least for my purposes. Comparing squirrel 1.x and lua is quite understandable, comparing Squirrel 2.x and Lua id a bit unfair because squirrel now packs many more features(especially types) that in a dynamically typed language means a lot of runtime checks. However means better performances in different fields. * Squirrel arrays are ways faster and more compact than tables used as arrays * Class instances use less than half memory of the equivalent solution done with tables(and we got alot of instances). * Weak references have been a bless from god in our project, in terms of resource management(we do resource management in squirrel). * Proper class inheritance allows to create a more comfortable programming model for the average C++ programmer. Also I think when looking at the performances of a language of this kind, we have to look at a real scenario, stuff like The great language Shootout is in my advice pointless. In a real app you have several megs of data living in the VM; what is going to affect your performances is not the bytecode speed (that's almost linear). Mostly, as you realized yourself , the performance problem will be memory. Memory management (if is a GCd language) plus memory aliasing and pagefaults and cache misses. Ref counting solves the first issue by giving a linear/predictable cost in memory management. Smaller and more cache friendly data structures improve the aliasing (my attempt to solve this in squirrel are classes and arrays). To give you an example in our engine we have written a very good memory manager that keeps blocks of memory of the same size in the same virtual page(we do not use a heap, we go straight to the os virtual page and take advantage of the 4Kb memory alignment) and squirrel performs 3 times faster on my memory stress test that tries to reproduce a game loop creating and deleting stuff. It makes the system behave like a language with a copy collector. To conclude, what I'm trying to say is that I wouldn't be too concerned if the bytecode execution speed is 30% slower in some tiny sorting test, when at the end in a real system that's not going to make any difference. Anyway I'm still working to improve squirrel performances.

Keep in mind many games use Python, which is very slow compared to Lua/Sq/As, however it's still fast enough for real-time games.

As for Squirrel vs. AngelScript, one would need to first determine how easy/complete the integration with C++ to script is. SqPlus (for Squirrel) does everything I need, quickly and easily. Perhaps someone with an unbiased opinion could review each of the scripting langauge binding systems and ease of use (Lua, AS, Sq, Python, C#, etc.).

From the benchmarks presented for AS vs. Sq, the AS C++ to script calls are so slow that it would appear something is wrong (23 times slower?). 3 of the tests are close, with 3 showing AS 3-6x faster. A real game/app would need to be tested with script-to-C++ and C++-to-script calls to see if there is any significant difference between the two languages. For systems with lots of classes, inheritance, and many megs of resource data in script, Alberto states that Sq is 3x faster than Lua^* (for his MMO project). More tests/benchmarks would be cool to see. ^*Estimate based on other comments vs. Lua.

For those looking for ultimate scripting speed, look no further than C#. It compiles very quickly to code that is not much slower than native C++. Using Mono or dotGNU, game related C# code should run anywere (not clear about PSP or other memory limited systems). For something like Alberto's MMO, Squirrel may still work better due to his customized memory allocator and Squirrel's design to keep memory related GC burst free.

Quote:

Original post by WitchLord Cool! [grin] This really shows me where I have to spend some time optimizing the code for AngelScript. :) I'll also take a look at that benchmark code you have, to see if you're doing everything the optimal way. ;) Regards, Andreas

1) Thank you Andreas. I hope this 'ugly bench startup' can be improved to became a real bechmark. I also started to profile lua (as seen in the code),
but I did not have time (I did it it at my job (cheating) in respiro times) to finalize it.


2) I use VC7.1
3) .. is interesting calling script to script taking less time than script to c/c++ and vicevera. I hope I did not make any mistakes in the code. You guys can doublecheck and any lua expert can finalize the lua part.

I am personally happy with either of them.
As any language you find advantages and disadvantages on both sides.
I hope the test are close to be accurate and I am waiting authors
reply if I did anything wrong (function caching, template wrappers etc.)
Thx....

Here are latest results with a more realistinc calls passing 2 params
into/and from script (time in microseconds). What is interesting
is that over time one of the libs (or both) eats virtual memory and
execution times increases exponentially.


and the scripts


[Edited by - Marius on May 28, 2006 11:08:34 PM]

You can reduce SqPlus overhead slightly by changing:



Part of your Squirrel 2.1 benchmark overhead is due to SqPlus: I have not yet made an optimization pass on the code. The ease of use and generality of template binding systems come at a small runtime overhead cost (template binding system overhead varies with implementation).

Squirrel/SqPlus should not leak any memory (I use it in my game; haven't seen any bug reports regarding memory leaks).

I have profiled them sapratelly. Angel call C++->Script increases.

AngelScript doesn't have any memory leaks that I'm aware of. But it does a lot of allocating/deallocating of memory which causes memory fragmentation, which in turn may very well be what is increasing the execution time with repeated execution.

I'll work to minimize the memory fragmentation in future versions.

Quote:

Original post by WitchLord AngelScript doesn't have any memory leaks that I'm aware of. But it does a lot of allocating/deallocating of memory which causes memory fragmentation, which in turn may very well be what is increasing the execution time with repeated execution. I'll work to minimize the memory fragmentation in future versions.

Squirrel+SqPlus also allocate and free memory (especially for instances). You might look at his benchmark and see if the code properly releases references when using AngelScript. I had a bug in SqPlus that did not properly restore the Squirrel stack that had behavior similar to what Marius described with AngelScript.

Indeed. I took a look at the benchmark code, and it seems that the asIScriptContext is never released after it is used. This would indeed eat a lot of memory and slow down the execution.

Fixing this should improve the overall performance measured in this benchmark. But my guess is that squirrel will still win with about 100 times the performance for calling script functions.

The difference is currently very large between the best and the worse performance for AngelScript:

10000 calls from C++ 2 script    S = 8089       A = 722240      0.0110000 calls from C++ 2 script    S = 7890       A = 5718446     0.00

All 10 executions should be approximately the same.

Another thing that can be improved is to reuse the context. Instead of creating a new context and then releasing it again for each script function that is called, you could create one context and use that for all the calls. Simply call Prepare() and then Execute() on the context.

Creating multiple contexts is only useful if you want to keep various scripts running in parallel, for example co-routines, or even multiple threads.

I see the diferences you have between scripts is quite big...
Here is an image taken from latest results I just take on my P4
winnt 512 Mb.



[Edited by - WitchLord on June 4, 2006 9:42:24 AM]

Your screenshot doesn't show up. I get a Gateway Timeout message when I try to view it in the browser.

Yeah, my computer is getting really old, it isn't very good at running even simple benchmarks any more. [wink] It's a Celeron 800MHz with 256MB of memory, running WinXP SP 2. It makes sense that I get a large difference in performance between Squirrel and AngelScript, after all the benchmark is allocating more than 100MB of memory for AngelScript without releasing it. [wink] (10*10000 calls, where each call allocates a context with 1KB stack space)

Regards,
Andreas

PS. I edited your post because it was missing an apostrophe in the image tag, making the page show up all wrong.

I ran some test on C-smile with tiscript.lib and tiscript.dll
(no source code for these libs)
Here are the results:
Because there is no method of holding on to a script
function by a handler or somekind of ID the call has to be explicit
done by function name. When calling a function from C to script
and viceversa the call time is proportional with the function name length.

The call of 'main'() C->c-smile function is ~47943 microseconds/10000 calls
and the call of 'mainmainmainmainmain'() is ~80943 microseconds/10000 calls

A call c-amile->c-smile of a void function receivineg an int is ~7609 microsecs/10000 calls

tested:
Dell OPTIPLEX GX260
P4 2.00 Ghz
1Gb Ram
compiler VC6, release mode non-optimized.

Are you going to update the benchmark program with the suggestions I made?

I just made a little test on my own (while optimizing the new version of AngelScript) and saw that reusing the context for each call, instead of creating a new context, gives a performance increase of about 10-15 times. You should get the same performance increase in the current version of AngelScript and it would be great if you could update your benchmark program to reflect that.

Especially since it may show that AngelScript is actually faster than Squirrel even with this trivial test of calling a void function. [wink] I'm basing this assumption on the first execution from your screenshot. Of course, it may just be wishful thinking.

Regards,
Andreas