Question

In my WinForms app I am loading a large amount of data from binary files into an SQLite DB. I do lookups on the DB to determine if each new piece of data is to be added e.g.

If LookUpResult Then
   AddNewData
Else
   DiscardNewData
End If

This is slow. The INSERTs seem to take a long time. Should I use DataTable or DataSet classes to load the data to RAM and then write to the SQLite DB as a background task? Or would inserting many rows at a time from a DataTable object be less costly?

I was previously loading all the data into a custom class and that was fast but likely to hit memory limits, hence the move to the DB. I would have thought SQLite would cache the INSERTs to RAM anyway but that doesn’t seem to be the case.

Thanks for any advice.

Answer 1

To ensure crash consistency, changed data is synchronized with the disk at the end of every transaction.
If you don’t use explicit transaction, each command will get its own automatic transaction, which implies that you get the synchronization overhead for each command.

Use one transaction for all your inserts.
(Your own changes will be visible, as long as you go through the same DB connection.)