Question

I have an application, which runs all the time and receives some messages (rate of them varies from several per second to none per hour). Every message should be put into a SQLite database. What’s the best way to do this?

Opening and closing the database on each message doesn’t sound good: if there are tens of them per second, it will be extremely slow.

On the other hand, opening the database once and just writing to it can lead to loss of data if the process unexpectedly terminates.

Answer 1

It sounds like whatever you do, you’ll have to make a trade-off.

If safety is your top-most concern, then update the database on each message and take the speed hit.

If you want a compromise, then update the database write every so many messages. For instance, maintain a buffer and every 100th message, issue an update, wrapped in a transaction.

The transaction wrapping is important for two reasons. First, it maximizes speed. Second, it can help you recover from errors if you employ logging.

If you do the batch update above, you can add an additional level of safety by logging each message as it comes to a file. You will reset this log every time a database update is successfully issues. That way, if an update fails, you know it failed on the entire block (since you are using transactions) and your log will have the information that did not update. This will allow you to re-issue the update, or even see if there was a problem with the data that caused the failure. This of course assumes that keeping a log is cheaper than updating the database, which can be the case depending on how you are connecting.