Contact caching

How are you searching the contact list? Box2D has a list of contacts per colliding body, so it usually only has to search through a few contacts to find a match. This could be a problem for a body with a large number of contacts, but if you are only using circles and they are all about the same size that won't be the case.

If you do use a hash table (and this is probably the easier method), watch out for memory allocations. In C++, std::unordered_map will allocate per entry, and this is slow.

There's more to making collisions go fast. I've written about it here:

http://dtosoftware.wordpress.com/