Question

I have an application in C++ which needs to record a large amount of data in a database. In general, this task is straight-forward whereby I can use any database-connector in c++ and get the job done.

However, I intend to separate the task from other critical path or basically, separate this utility from the main application. Can you suggest an efficient way? Or writing to the DB is itself efficient/safe enough?

I am thinking of writing the data to some file and running a separate script in the back-ground to dump the data appended to the file in the database. Is this a reasonable approach? Or is writing to a file equally inefficient/unsafe as writing to a database?

Thanks

Answer 1

The question you need to ask yourself is:

Do I need durability?

Durability is the aspect of ACID transactions and it essentially means that when DBMS signals to the client that the transaction has committed, the changes made by that transactions are guaranteed to be persistent. So even if there is a power outage immediatelly after commit, the data is safe.

• If the answer is “no”, then you are free to spin-off a background thread that will write to the database, possibly long after the “main” processing has finished.
• If the answer to that question is “yes”, then you have no choice but to do your insert “in place”, or at the very least make sure the background thread has finished before you finish the “main” processing.

In any case, properly binding parameters and preparing your INSERT statements should help the performance.

If you go the background thread route, you’d probably want to pass the data as it becomes available via a message queue, so the writing to the database can happen in parallel with your “main” processing.