I have a sqlite database with only one table (around 50,000 rows) and I recurrently perform update-otherwise-insert operations on it using Java and sqlitejdbc (i.e. I try to update rows if they exist and insert new rows otherwise). My table is similar to a word frequency table with “word” and “frequency” columns, and without a primary key!
The problem is that I perform this update-otherwise-insert operation hundreds of thousands of times and on average the insert or update operation takes more than 2ms. There are even times when the insert operations take some 20 milliseconds. I should also mention that the table has an index on the column on which I use the “where” clause in my insert operations (the “word” column”), which naturally should make the insert operation more expensive.
Firstly I want to make sure if 2ms for an insert operation on an indexed table with 50,000 rows is normal and there isn’t anything that I’ve missed, and after that any suggestions to improve the performance is more than welcome. It struck me that dropping the index before performing large crunches of insert operations and recreating it again afterwards is good practice, but I can’t do it here because I need to check if a row with the same word already exists.
I know all the stuff about “it depends on the hardware” and “it depends on the rest of your code” etc, but I really think one CAN have an idea of how much an insert operation should take on an average pc.
I partially solved my problem. For anyone interested in an answer to this, this link will be helpful. In short, turning off journal mode in sqlite (“pragma journal_mode=OFF”) improves the insert performance significantly (almost four times the previous speed in my case) to the cost of making the code prone to data loss in case of unexpected shutdown.
As for the normal insert speed, it is way faster than 2ms/operation. It can reach as high as hundreds of thousands of insert operations per second using the right pragma instructions, making best use of transactions, etc.