🔒 Locked

Presenting: Memory manager/tracker

Started by Jan Wassenberg Jan 20, 2005 at 12:40 PM 34 replies 11.1k views

Original Post

Jan 20, 2005 at 12:40 PM

I am making available under the GPL a portable memory manager/tracker for C and C++ that can spot the following problems: memory leaks, double-free, allocation over/underruns, unused memory, and use-after-free. Basically, it is a more capable and customizable replacement for VC's debug heap and/or the memory part of BoundsChecker. Download at http://www.stud.uni-karlsruhe.de/~urkt/mmgr.zip ; see documentation in mmgr.h . Installation: the newest MS debugging tools are required; download and copy dbghelp.* into the appropriate compiler directories. The mmgr code (~1400 lines) is fairly clean and documented by now, but isn't yet finished, i.e. bulletproof and armor-plated. In particular, a good deal of dependencies on my personal library remain; the headers needed to compile are included. Bug reports, suggestions, and comments are very welcome. Please do make sure, however, that you aren't reporting bugs in your own application! ;) It is my hope that this release helps someone avoid the pain of tracking down hard-to-find memory corruption. Pay it forward! // edit: added link to the now mandatory debug tools [Edited by - Jan Wassenberg on February 21, 2005 1:03:36 PM]

okonomiyaki

Jan 21, 2005 at 10:22 AM

Interesting. I'm currently reviewing options for a memory manager in our current project, so it's nice to have options. Looks good, I'll have to try it out.

I'd recommend making a readme file though, instead of putting the main documentaion inside the header of a file.

Also, any chance for it to support pooling sometime? And will you be frequently updating it?

rick_appleton

Jan 21, 2005 at 10:41 AM

Always nice to see people opening such projects up to the public.

However I have a few questions (note that I haven't downloaded the file, so if the answers are in there, I'm sorry):
1. Is this platform dependent?
2. What does your manager have over the one downloadable from www.fluidstudios.com?

Skizz

Jan 21, 2005 at 10:53 AM

Quote:
memory leaks, double-free, allocation over/underruns, unused memory, and use-after-free

Hang on, doesn't the MSVC debug run time do all that already?
Hint: look up _CrtSetDbgFlag.

Skizz

Jan Wassenberg

Jan 21, 2005 at 12:45 PM

Thanks for all comments.

okonomiyaki:

Quote:
I'd recommend making a readme file though, instead of putting the main documentaion inside the header of a file.

hmm, external readme files tend to get out of sync with reality ;) I prefer even the high-level documentation to reside in header files - everything in one place.

Quote:
And will you be frequently updating it?

Yes, it is part of the library code used throughout all of my projects (including 0 A.D.), so it will be updated frequently.

Quote:
Also, any chance for it to support pooling sometime?

I have thought about completely replacing the CRT memory allocation routines with more efficient versions, since 0ad is quite memory-heavy. That's no small undertaking, though - it's on the back burner for now.
Is this (adding e.g. a pool allocator) what you meant?

rick appleton:

Quote:
1. Is this platform dependent?

Platform independence was a design goal, but I can only test on Win32. I would be grateful for reports of compilation on other platforms, or problems encountered in so doing.

Quote:
2. What does your manager have over the one downloadable from www.fluidstudios.com?

It is derived from the Fluid Studios code, but completely overhauled. In particular, it has been modularized (with lots of duplicated code removed) and made thread-safe.

Skizz:

Quote:
Hang on, doesn't the MSVC debug run time do all that already? Hint: look up _CrtSetDbgFlag.

Indeed; its interface influenced my design. My mmgr needs to work on other platforms, though (we also develop on *nix). It also goes a little further than the debug heap, adding detailed allocation logging, random induced failures, and unused memory tracking.

okonomiyaki

Jan 21, 2005 at 2:15 PM

Quote:

I have thought about completely replacing the CRT memory allocation routines with more efficient versions...

Yep, that's what I meant basically. I was thinking more strictly to the technique of allocation large chunks of memory and feeding them out to the application, but I'm sure that by saying "more efficient versions" you meant things like that :) I'll keep it bookmarked!

rick_appleton

Jan 22, 2005 at 3:58 AM

Quote:

Quote:
1. Is this platform dependent?

Platform independence was a design goal, but I can only test on Win32. I would be grateful for reports of compilation on other platforms, or problems encountered in so doing.

Quote:
2. What does your manager have over the one downloadable from www.fluidstudios.com?

It is derived from the Fluid Studios code, but completely overhauled. In particular, it has been modularized (with lots of duplicated code removed) and made thread-safe.

Ah, ok then. Thank you for the info. So it offers all the functionality of that code (writing to log file etc, and breaking on allocation errors)? I only recently realized that the FluidStudios mmgr isn't cross-platform, so I will definately be trying this out on Linux.

Edit: I just realized that you're releasing under the GPL :(. Because placing my projects under GPL is not acceptable to me I have two questions:
1. Why did you choose GPL as licence and not LGPL?
2. What happens if I use the mmgr in my own personal code, but remove it from the builds I release to the public. If it works like the FluidStudios manager, it's more a tracker than a manager, and it can easily be compiled out. This makes this question pretty important I guess.

[Edited by - rick_appleton on January 22, 2005 5:58:04 AM]

Jan Wassenberg

Jan 22, 2005 at 6:38 AM

Quote:
Yep, that's what I meant basically. I was thinking more strictly to the technique of allocation large chunks of memory and feeding them out to the application, but I'm sure that by saying "more efficient versions" you meant things like that :) I'll keep it bookmarked!

Righto :) I think I will in time do this, but it won't be in the next few weeks (busy).

Quote:
Ah, ok then. Thank you for the info. So it offers all the functionality of that code (writing to log file etc, and breaking on allocation errors)?

Yes.

Quote:
I only recently realized that the FluidStudios mmgr isn't cross-platform, so I will definately be trying this out on Linux.

OK, please let me know if it compiles :)

Quote:
1. Why did you choose GPL as licence and not LGPL?

In the hope that more source code would be released, so all may benefit :)

Quote:
2. What happens if I use the mmgr in my own personal code, but remove it from the builds I release to the public. If it works like the FluidStudios manager, it's more a tracker than a manager, and it can easily be compiled out. This makes this question pretty important I guess.

I'm no lawyer, but if you are very careful to strip it out in every release build, you should be ok :)
hehe, but you lose if/when I add a more efficient malloc implementation. Why is the GPL not "acceptable" for you?

rick_appleton

Jan 22, 2005 at 7:21 AM

Well, basically I'm working on my framework (which will be cross platform) and I am planning to use that to create games with. And I am very tempted to try and sell these games over the internet as an indie developer. Which would be pretty difficult if the source code were freely available.

Regarding the compilation, under Linux I get the following errors. I will test it on Windows this afternoon and see if I get the same results there:
I'm getting errors on calls to delete:

char *d = new char[90];delete [] d;

gives the following error on the second line:

error: `((d != 0) ? operator delete [](void*)(d) : 0)' has type `void' and is not a throw-expression

and the following

char *d = new char;delete d;

gives on the second line:

operator delete(void*)(d)' has type `void' and is not a throw-expression

vNistelrooy

Jan 22, 2005 at 7:25 AM

I've already coded a memory manager, completly replacing the standard one. All your features plus a few more.

rick_appleton

Jan 22, 2005 at 8:34 AM

I'm getting lots of errors in sysdep\win\wsock.h and sysdep\win\wposix.h when compiling under Visual C++ Express Beta.

Do you use a chat client? If you want to add me, it'll make discussing stuff like this easier (msn: rick_apREMOVETHISpletonAThotmailDOTcom). The GameDev irc channel is also okay for me.

Jan Wassenberg

Jan 22, 2005 at 10:47 AM

Quote:
Well, basically I'm working on my framework (which will be cross platform) and I am planning to use that to create games with. And I am very tempted to try and sell these games over the internet as an indie developer. Which would be pretty difficult if the source code were freely available.

Gotcha. I used to think the same way, but I figure most users don't even know what a compiler is, much less how to successfully grab your code from CVS and make it into a functional game (heh, unless you get your act together, compiling any large project is pretty much a miracle even for devs ;P). That, and the common approach of copyrighting / not making your data files available protects you.

[don't take this personally, it's not directed at you:] But that aside, why should anyone be able to take advantage of open source software for their own profit, without giving anything back? GPL is all about share and share alike; for there to be anything useful for someone to reuse, someone must already have taken the "I will not profit monetarily from this" plunge (in the case of standalone libraries).
As Michael Abrash says: "None of us learn in a vacuum; we all stand on the shoulders of giants such as Wirth and Knuth and thousands of others. Lend your shoulders to building the future!"

Quote:
Regarding the compilation, under Linux I get the following errors. I will test it on Windows this afternoon and see if I get the same results there:
I'm getting errors on calls to delete: [..] error: ((d != 0) ? operator delete [](void*)(d) : 0)' has type void' and is not a throw-expression

Thanks for the report! My test platform is VC7.1 and ICC 8.0. I presume GCC isn't liking the delete #define, which is evil. (snip rant about the poor design of C++ mem allocation)
Am investigating solutions; I was thinking of determining the caller via asm return address trickery, instead of macros.

vNistelrooy:

Quote:
I've already coded a memory manager, completly replacing the standard one. All your features plus a few more.

What is this? Trolling? Bragging?
Unless you have something to say that brings us forward, you may profitably be silent.

Quote:
I'm getting lots of errors in sysdep\win\wsock.h and sysdep\win\wposix.h when compiling under Visual C++ Express Beta.
Do you use a chat client? If you want to add me, it'll make discussing stuff like this easier (msn: rick_apREMOVETHISpletonAThotmailDOTcom). The GameDev irc channel is also okay for me.

VC express? uhoh ;) A quick fix for those headers: you don't need the contents of wsock at all, and everything except pthread_mutex* in wposix.h should be expendable. I'd be happy to discuss this on IM, and will contact you ASAP.

Jan Wassenberg

Jan 27, 2005 at 12:03 PM

Major update, download address remains the same.
- caller information is no longer stored in globals. This couldn't handle destructor chains and wasn't foolproof (incorrect owner could be reported). Instead, the call stack is examined and caller is determined via debug information (implemented for Win32, pending for *nix).
Major advantage: allocations via global operator new() made from STL or other external libraries can now be tracked.

- many improvements to low-level code; less dependencies.

- stack trace whenever anything goes wrong; outputs local variables and their types+values.

When compiling for Win32, the newest debugging tools are required (updates dbghelp.* to v6.3).

rick appleton: haven't seen you online yet. Interop problems with Trillian? You can drop me a line at ICQ 135709088 or MSN janwassenberg@gmx.net (@spambots: harvest away! that's my very own spam dump).

griminventions

Feb 4, 2005 at 1:41 PM

Nice, I'm gonna give this a try. :)

griminventions

Feb 4, 2005 at 3:50 PM

Hmm, I get

error LNK2019: unresolved external symbol "void __cdecl operator delete(void *,char const *,int,char const *)"

for delete and new in several places. The lib is in the link path, but what else could it be? I'm using VC7.1 with the latest MS debugging tools installed.

Aldacron

Feb 5, 2005 at 12:41 AM

It's great of you to do this. This sort of thing could be useful for a lot of people around here. I think it would be more useful if it weren't GPL, though :)

Quote:
Original post by Jan Wassenberg
But that aside, why should anyone be able to take advantage of open source software for their own profit, without giving anything back? GPL is all about share and share alike; for there to be anything useful for someone to reuse, someone must already have taken the "I will not profit monetarily from this" plunge (in the case of standalone libraries).
As Michael Abrash says: "None of us learn in a vacuum; we all stand on the shoulders of giants such as Wirth and Knuth and thousands of others. Lend your shoulders to building the future!"

Share and share-alike is a great philosophy, but don't you feel that sharing should be unconditional? Using the GPL is like saying, "I will share with you but only on the condition that you share with others". If someone back in my elementary school cafeteria told me that he would share his sandwich with me only if I shared my milk with our friend Joe, I likely would decline if I were thirsty. It would be a bit friendlier to say that he'll share his sandwich with me and that I ought to share my milk with Joe if I'm not too thirsty.

Open-source works best, I think, when I can use it in both my open source projects and my closed source projects. That should be my decision. As I see it, there are very few GPL projects out there that I would open up the source of a commercial game to use (Quake 2 engine being one of them). Something like this, while a boon to the community, wouldn't warrant that in my opinion.

I respect your decision though, and I always appreciate when people open their source. I also don't want to incite a religious war. I'm just giving you my view in the hope that you reconsider the license :) I think licenses like BSD, MIT, ZLIB, and LGPL (where it makes sense) are more preferable. GPL is too draconian.

Jan Wassenberg

Feb 8, 2005 at 7:30 AM

Hope Mardi Gras/Fasching was good for y'all.

griminventions:

Quote:
Hmm, I get
error LNK2019: unresolved external symbol "void __cdecl operator delete(void *,char const *,int,char const *)"
for delete and new in several places. The lib is in the link path, but what else could it be? I'm using VC7.1 with the latest MS debugging tools installed.

Uhoh, not good! ;) Various overloads of operator new and delete are declared in mmgr.h and defined in mmgr.cpp.
Have you perchance changed the default calling convention? Is mmgr.h included from every single header? (I recommend putting it in the PCH, so the compiler will warn you if not included)
I assume since you don't get any other linker errors that mmgr.cpp is actually in the project. Is this only happening for that specific overload (file,line,function) of new/delete, or others as well?

Aldacron:

Quote:
It's great of you to do this. This sort of thing could be useful for a lot of people around here. I think it would be more useful if it weren't GPL, though :)

Thanks, I'm glad to.
heh, another GPL vs. BSD licensing discussion. Oh well, here goes.. ;) We see eye-to-eye if you mean short-term usefulness for many projects here. However, in the long term, my concern is that improvements made to the code if under a 'liberal' license (BSD, MIT, ZLib) could be folded under a proprietary license. The GPL ensures this code and any future improvements remain free, to the benefit of all. This is in addition to the issue below.

Quote:
Share and share-alike is a great philosophy, but don't you feel that sharing should be unconditional? Using the GPL is like saying, "I will share with you but only on the condition that you share with others". If someone back in my elementary school cafeteria told me that he would share his sandwich with me only if I shared my milk with our friend Joe, I likely would decline if I were thirsty. It would be a bit friendlier to say that he'll share his sandwich with me and that I ought to share my milk with Joe if I'm not too thirsty.

In a perfect world, sharing would indeed be unconditional. Since ours isn't, I cover the misgivings of making my work freely available by requiring that anyone who uses it be of the same spirit.
I would, after all, be reluctant to help someone who is only looking out for his own gain. In your analogy, the 'friendlier' way assumes you would actually in turn share your milk if you could. Since that's not always the case, I prefer GPL to LGPL (which would only solve the issue of freedom of modifications to the code).

Quote:
Open-source works best, I think, when I can use it in both my open source projects and my closed source projects. That should be my decision. As I see it, there are very few GPL projects out there that I would open up the source of a commercial game to use (Quake 2 engine being one of them). Something like this, while a boon to the community, wouldn't warrant that in my opinion.

Agreed. Many large games (including 0 A.D., the original benefactor of this code) cannot be open-sourced in entirety as the GPL would require, due to security concerns. Therefore, I am willing to entertain discussion of a second license in return for other considerations :)

Quote:
I respect your decision though, and I always appreciate when people open their source. I also don't want to incite a religious war. I'm just giving you my view in the hope that you reconsider the license :) I think licenses like BSD, MIT, ZLIB, and LGPL (where it makes sense) are more preferable. GPL is too draconian.

Well said.
However, as explained above, I think that for the moment my intent is well-served with a GPL license. For projects that can't go GPL, I would consider a dual-license; another alternative would be to use the manager only during internal tests (as outlined in a previous post).

griminventions

Feb 9, 2005 at 11:10 AM

Quote:
Original post by Jan Wassenberg
Have you perchance changed the default calling convention?

No.

Quote:
Is mmgr.h included from every single header?

It's in every implementation file, but not every header file.

Quote:
I assume since you don't get any other linker errors that mmgr.cpp is actually in the project.

It wasn't before, but I put it there after reading this and the new/delete unresolved links are gone. But some new ones arrived...

mmgr.obj : error LNK2019: unresolved external symbol "int __cdecl debug_assert_failed(char const *,int,char const *)" (?debug_assert_failed@@YAHPBDH0@Z) referenced in function "unsigned int __cdecl mmgr_set_options(unsigned int)" (?mmgr_set_options@@YAII@Z)mmgr.obj : error LNK2019: unresolved external symbol "int __cdecl pthread_mutex_lock(void * *)" (?pthread_mutex_lock@@YAHPAPAX@Z) referenced in function "void __cdecl lock(void)" (?lock@@YAXXZ)mmgr.obj : error LNK2019: unresolved external symbol "int __cdecl pthread_mutex_unlock(void * *)" (?pthread_mutex_unlock@@YAHPAPAX@Z) referenced in function "void __cdecl unlock(void)" (?unlock@@YAXXZ)mmgr.obj : error LNK2019: unresolved external symbol "void __cdecl debug_check_heap(void)" (?debug_check_heap@@YAXXZ) referenced in function "bool __cdecl mmgr_are_all_valid(void)" (?mmgr_are_all_valid@@YA_NXZ)mmgr.obj : error LNK2019: unresolved external symbol "int __cdecl debug_resolve_symbol(void *,char *,char *,int *)" (?debug_resolve_symbol@@YAHPAXPAD1PAH@Z) referenced in function "void * __cdecl alloc_dbg(unsigned int,enum AllocType,char const *,int,char const *,unsigned int)" (?alloc_dbg@@YAPAXIW4AllocType@@PBDH1I@Z)mmgr.obj : error LNK2019: unresolved external symbol "void * __cdecl debug_get_nth_caller(unsigned int)" (?debug_get_nth_caller@@YAPAXI@Z) referenced in function "void * __cdecl alloc_dbg(unsigned int,enum AllocType,char const *,int,char const *,unsigned int)" (?alloc_dbg@@YAPAXIW4AllocType@@PBDH1I@Z)mmgr.obj : error LNK2019: unresolved external symbol "int __cdecl pthread_mutex_destroy(void * *)" (?pthread_mutex_destroy@@YAHPAPAX@Z) referenced in function "void __cdecl lock_shutdown(void)" (?lock_shutdown@@YAXXZ)mmgr.obj : error LNK2019: unresolved external symbol "int __cdecl pthread_mutex_init(void * *,void const *)" (?pthread_mutex_init@@YAHPAPAXPBX@Z) referenced in function "void __cdecl lock_init(void)" (?lock_init@@YAXXZ)

Thanks for your help. :)

Jan Wassenberg

Feb 9, 2005 at 3:25 PM

Quote:
>Is mmgr.h included from every single header?
It's in every implementation file, but not every header file.

Oops, that's what I meant. BTW, any headers containing inline code that allocates memory should also #include "mmgr.h".

Quote:
It wasn't before, but I put it there after reading this and the new/delete unresolved links are gone. But some new ones arrived...

Ah, ok. This is easy to fix: you just need to add all the files in the distribution to your project (with the possible exception of precompiled.*). Can't leave any out :)

evolutional

Feb 9, 2005 at 3:42 PM

This sounds pretty good. I was using the fluid studios manager myself, but have recently switched to Linux dev so this could be a real plus for me.

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.