Linux server process crashing after ~8 hours

Does the crashed process leave a core dump?

What SiCrane said. If it does not leave a core dump, learn how to run it so it does. You can also run your server from gdb, or attach gdb to the running program before it crashes.

Besides that, check if the process is growing in size, which would mean you have a memory leak of some sort.

It's also holding two ofstream open.

Like the others said your best bet is to get a core dump file for the application. You can then step through the core dumb with gdb to find where your crash is happening. Since it is a 8hr run odds are you have a memory leak somewhere as if it was a null pointer being accessed it would probably crash much much sooner. Read the man pages for core as it will explain how you use it if you are not getting actual core dumps. The general gist of coredumps is you can set a dumpfile and start your daemon and when it crashes it will write the report to that file. Sadly it is really the only way on linux to solve the problem besides invoking the daemon attached to GDB.

There is actually another option besides the above but I don't know how useful it will be for this situation. The command is called strace.

You can run strace -o /path/to/output.txt /path/to/daemon

The command will log everything that application does including the crash. It would give you reason for the crash but not the exact execution point of the crash.

Since it crashes over time, it does indeed sound like a slow memory leak.

My suggestion is to compile everything with '-g' (in GCC) to add debugging symbols.

Then run it in "valgrind --tool=memcheck" (Which you should really do before the output leaves the development machine anyway).

This should then tell you all allocations that are not subsequently deallocated so you should be able to pinpoint the issue pretty easily.

It just crashed with valgrind and it didn't say there were any memory leaks.

It seems it crashes whenever my open SSH session gets closed by me closing the laptop or leaving it on until it goes to sleep.

I thought I had turned on core dump following these instructions

http://www.cyberciti.biz/tips/linux-core-dumps.html

but I don't see anything in /tmp.

So to run it under gdb I type in

gdb myProgram
run

But since I'm doing this remotely from an SSH shell I need it to fork the process and output to a file. How do I do that? I need it to continue running after I end the SSH session.

I can't just do

gdb myProgram &

because I need to type in "run" to gdb.

Oh, that's it. whenever I close the SSH session it crashes.

I thought it was enough to add "&" to fork the process.

How do I get it to continue running after I quit SSH?

Oh, figured it out

http://en.wikipedia.org/wiki/Nohup

Even though you found a solution, you might want to look into either tmux or [gnu]screen. They are basically terminal multiplexers that allow you to keep your session alive when you disconnect and then you can reattach to your previous session again by typing e.g "tmux attach".

Very handy, especially if you develop the software itself on a remote server.