I have a pretty big issue right now with a dead lock in a multi threaded software. I know which of my threads and mutexes cause the deadlock but I do not understand why.
I have a setup that looks like this (Pseudo c++ code)
AutoLock is a scoped lock class, nothing fancy.
class Foo
{
public:
void start(); //Spawns the thread that will call run()
void doStuffA() {AutoLock lock(&mutexA); <DoStuffHere()> }
void doStuffB() {AutoLock lock(&mutexB); <DoStuffHere()> }
private:
void extraWork()
{
AutoLock lock(&mutexB);
//Processing here
doStuffB();
}
void run() //Threaded function
{
while(true)
{
AutoLock lock(&mutexB); //Lock the B Mutex
//Do a lot of work, networking stuff, etc
extraWork(); //This is fine since we're using a recursive mutex
AutoLock lock2(&mutexA); //Dead lock here after a couple of hours of run time.
}
}
boost::recursive_mutex mutexA;
boost::recursive_mutex mutexB;
};
int main()
{
Foo foo;
foo.start();
while(true)
{
foo.doStuffA();
//Do stuff
foo.doStuffA();
//Do stuff
foo.doStuffA();
//Do some stuff
foo.doStuffB(); //Usually hangs here a bit while foo finishes a loop
}
}
So the code is obviously not exactly like this but the logic is the same. We ran this code with no problems for a long time and it just recently starting deadlocking. We traced the code with dumps and know this setup is causing the problem.
Note that the main thread cannot lock A or B mutex directly but only by calling public functions of Foo. Since we lock only using the AutoLock class (Scoped lock), the main thread should never keep a lock on the mutexes. Yet, the thread sometimes hangs indefinitely when trying to lock mutex A.
I know from looking at boost code that the hanging only happens if the current thread id inside the mutex is different from the one calling ->lock(). Therefore there's only 2 explanations to this problem.
1. The main thread somehow keeps a lock on mutex A.
2. There's memory corruption that messes the data of the mutex A.
I'm really out of ideas and if some multithreading guru could give me tips on what to look for it would be greatly appreciated.
