I'm currently writing a script sanbox with AngelScript, but I don't understand how to limit the memory used by the scripts.
I've looked at asSetGlobalMemoryFunctions, but I'm getting stuck on writing the free function :
size_t g_allocatedMem = 0; // memory currently allocated. Must be 0 at program end.
size_t g_nbAllocations = 0; // ++ on alloc, -- on dealloc. Must be 0 at program end.
In order to know how much is deallocated you'll need to keep track of how much was allocated in one way or another.
If you don't care much about portability I'm pretty sure the malloc() function stores this in the allocated block of memory, by examining the bytes just before the returned pointer you'll probably find the size there. This will of course depend on the implementation that the compiler vendor provides. But if you value portability you should do it yourself.
The easiest is to simply allocate a few extra bytes with each allocation and store the size of the allocated block in those bytes. Another option is to keep an associative map with the pointer is the key and the size is the value.
In the test_feature project that I use to test the library I've implemented a memory manager like this:
//#define TRACK_LOCATIONS
//#define TRACK_SIZES
static int numAllocs = 0;
static int numFrees = 0;
static size_t currentMemAlloc = 0;
static size_t maxMemAlloc = 0;
static int maxNumAllocsSameTime = 0;
static asQWORD sumAllocSize = 0;
// Remember the size of the memory allocated at this pointer
memSize.insert(map<void*,size_t>::value_type(ptr,size));
// Remember the currently allocated memory blocks, with the allocation number so that we can debug later
memCount.insert(map<void*,int>::value_type(ptr,numAllocs));
// Determine the maximum number of allocations at the same time
if( numAllocs - numFrees > maxNumAllocsSameTime )
maxNumAllocsSameTime = numAllocs - numFrees;
#ifdef TRACK_SIZES
// Determine the mean size of the memory allocations
map<size_t,int>::iterator i = meanSize.find(size);
if( i != meanSize.end() )
i->second++;
else
meanSize.insert(map<size_t,int>::value_type(size,1));
#endif
#ifdef TRACK_LOCATIONS
// Count the number of allocations for each location in the library
loc l = {file, line};
map<loc, int>::iterator i2 = locCount.find(l);
if( i2 != locCount.end() )
i2->second++;
else
locCount.insert(map<loc,int>::value_type(l,1));
#endif
return ptr;
}
void MyFreeWithStats(void *address)
{
// Count the number of deallocations made
numFrees++;
// Remove the memory block from the list of allocated blocks
map<void*,size_t>::iterator i = memSize.find(address);
if( i != memSize.end() )
{
// Decrease the current amount of allocated memory
currentMemAlloc -= i->second;
memSize.erase(i);
}
else
assert(false);
// Verify which memory we are currently removing so we know we did the allocation, and where it was allocated
map<void*,int>::iterator i2 = memCount.find(address);
if( i2 != memCount.end() )
{
// int numAlloc = i2->second;
memCount.erase(i2);
}
else
assert(false);
printf("---------\n");
printf("MEMORY STATISTICS\n");
printf("number of allocations : %d\n", numAllocs); // 125744
printf("max allocated memory at any one time : %d\n", (int)maxMemAlloc); // 121042
printf("max number of simultaneous allocations: %d\n", maxNumAllocsSameTime); // 2134
printf("total amount of allocated memory : %d\n", (int)sumAllocSize); // 10106765
printf("medium size of allocations : %d\n", (int)sumAllocSize/numAllocs);
#ifdef TRACK_SIZES
// Find the mean size of allocations
map<size_t,int>::iterator i = meanSize.begin();
int n = 0;
int meanAllocSize = 0;
while( i != meanSize.end() )
{
if( n + i->second > numAllocs / 2 )
{
meanAllocSize = (int)i->first;
break;
}
n += i->second;
i++;
}
printf("mean size of allocations : %d\n", meanAllocSize);
printf("smallest allocation size : %d\n", meanSize.begin()->first);
printf("largest allocation size : %d\n", meanSize.rbegin()->first);
printf("number of different allocation sizes : %d\n", meanSize.size());
Thanks for your quick answer.
Actually, I forgot to call cleanup functions, now everything is ok about memory leaks.
I think that adding extra-data to memory blocks will be better for portability.
I had also imagined to first allocate a big memory block the size of the memory limit (not so much),
and then using it to store script's data, taking care not run out of it... but I don't know how to manage this.
In my project, every user will be able to send scripts to a server, that will run them (each in separated sandboxes),
so I have to secure the script manager a lot to avoid spamming.
Also, are the scriptarray or scriptstdstring addons using the same alloc functions? Or do I have to modify them?
I will examine your code, it will be very useful for the moment
The add-ons are not using the memory allocation functions that the engine do. If you want them to allocate memory through the same routines you'll have to modify them yourself.
I've successfully implemented the custom alloc and free functions, but now I have a new problem :
When I reach the memory limit (actually just before), how do I inform the program to stop the sandbox without crash?
Because I'm currently returning 0 (out of memory) in memAlloc, but then my program gets a SIGSEV and closes immediately.
Should I throw an exception instead?
My code :
void * Sandbox::memAlloc1(size_t size)
{
// Check memory used
m_state.currentMemUse += size;
if(m_state.currentMemUse >= m_state.memLimit)
{
return 0; // out of memory
}
Can you show me where the SIGSEV is occurring? i.e. the callstack?
Returning 0 is the right way as I'm not using exceptions at all in AngelScript, but I must confess that AngelScript probably doesn't handle out-of-memory situations correctly in all locations. I'll have to fix these as they are found out.
I added some stuff to the script (one class with 2 attributes and 2 methods) and set the limit to 65000.
SIGSEV occured again as the new limit has been reached :
Thanks I'll have a look at this and see how to best handle the out-of-memory condition. I'll also go through the code and look for as many problems as possible.
I have added my custom memory functions to scriptarray, seems to work fine.
But when I try to create a too large buffer, I get a SIGSEV too.
I realized that if we run out of sandbox's memory, AngelScript wouldn't have space to allocate for errors handling.
So I've added a condition in memFree : if memLimit is reached, memFree will return 0 only once, and will not check the limit after (because the program using the sandbox is assumed to have stopped).
I recompiled and retried my test :
When the big buffer is created, the context exception is set to "Out of memory", the script stops.
Then the console prints "An exception 'Out of memory' occurred. Please correct the code and try again."
But when the engine is released, I get SIGSEV anyway :
I've made several improvements to the code for handling out-of-memory situations. You can get those improvements from the SVN, specifically revisions 1318 and 1319.
Out of memory situations are extremely difficult to test though so I really can't guarantee that there won't be any more SIGSEV problems, or memory leaks due to lack of clean-up due to 'out-of-memory'. I'll have to fix these problems as they are detected.
Let me know if you find any further problems.
I do suggest you avoid setting a hard limit for the memory consumption. It is a very severe condition and as you already noted, it will make it difficult to get proper error messages, and doing a proper cleanup of the script that uses too much memory. Instead of using the hard limit and returning 0 on the allocation routines, I suggest you monitor how much memory is used, and if the memory usage exceeds a threshold, you abort the script execution (with asIScriptContext::Abort), and then do the normal cleanup as if shutting down the script.
