Skip to main content
GameDev.net gamedev.net
🔒 Locked

dB for logging?

Started by Prune Sep 14, 2010 at 10:06 PM 2 replies 750+ views
Original Post
Prune
Prune
If I'm already using SQLite in a project, could there be any benefit of using the database for logging instead of a simple file logging? I guess I'm thinking if there's any way to leverage the recovery mechanisms of a database's storage engine in case of a crash, whereas if you're using a file you have to explicitly flush to drive after every message that exceeds a threshold where you'd be expecting an imminent crash.
Quite possibly it's a stupid idea, so be gentle in your replies...
"But who prays for Satan? Who, in eighteen centuries, has had the common humanity to pray for the one sinner that needed it most?" --Mark Twain

~~~~~~~~~~~~~~~Looking for a high-performance, easy to use, and lightweight math library? http://www.cmldev.net/ (note: I'm not associated with that project; just a user)
krez
krez
SQLite is ACID compliant, so basically it does the flushing for you. This only guarantees that it is done writing before it tells you it is done; a properly timed crash could cause you to lose something that you tried to write, but it won't tell you it is written if it isn't. Of course, this isn't any worse that just writing to a file (it is a bit slower).

One benefit of logging to a database is that you can use queries to easily filter your logs instead of digging through a humongous text file. Also, if you want to get fancy, you can easily set up your log to be more query-friendly; you can add in some other columns so you don't have to go with searching through text. I'm a big fan of "log level" (debug, info, warning, error), "function name" (if you are catching exceptions or something), a time stamp, etc etc, plus some text error message. That makes it much easier to find what you want, than to search through a wall of text.
--- krez ([email="krez_AT_optonline_DOT_net"]krez_AT_optonline_DOT_net[/email])
Prune
Prune
I do use log levels and timestamp/function/line/thread name/file decoration, and for levels exceeding thresholds, I write out the asynchronous log buffer to file (it also auto-writes out periodically). But since I'm using C++ stream i/o and there's no portable way to force a flush to disk with an fstream (I'm targeting Windows and Linux), one thing I'm hoping might be possible is to ask the database to sync from within the program (I've read that automatic sync is best disabled for performance).
"But who prays for Satan? Who, in eighteen centuries, has had the common humanity to pray for the one sinner that needed it most?" --Mark Twain

~~~~~~~~~~~~~~~Looking for a high-performance, easy to use, and lightweight math library? http://www.cmldev.net/ (note: I'm not associated with that project; just a user)
krez
krez
I don't think you can have it both ways. I believe with SQLite, you can either wait on a flush for each transaction, or use async mode and lose your ACID.

Have you considered wrapping groups of inserts in transactions? A crash would lose anything not committed yet, but you would get significantly better performance than having a whole slew of tiny transactions. You could issue a BEGIN TRANSACTION, and write whatever you want to your log. Then send a COMMIT periodically, and also any time you have something that MUST be written immediately. After every COMMIT, open a new transaction, and you can keep on loggin'.

You could also catch exceptions at some high enough level that pretty much any crash is caught, and issue a final COMMIT from there.

OR... SQLite is open source, so you could always hack your own force-flush function into async mode.
--- krez ([email="krez_AT_optonline_DOT_net"]krez_AT_optonline_DOT_net[/email])

Topic Locked

This topic has been locked by a moderator. New replies are not allowed.

Sign in to reply to this topic.