Question

Without getting into too much detail, I have a process in my iOS app that parses XML and inserts (sometimes up to 10s or 100s of) thousands of records into an Sqlite3 table.

For speed, I have no indexes on the table, and I’m inserting records into an in memory temporary table first then moving them over to the final table in batches of 400. Neither the temporary table or filesystem table have any indexes.

The problem I’m noticing is that at the beginning of the process, it takes about a 10th of a second to flush the 400 records to disk. Then with each flush, it takes longer and longer, where after a minute or two, it’s taking 3 seconds to flush the 400 records. The longer it goes, the longer it takes. Each flush takes about a 1 to 2 tenths of a second longer than the previous one.

Since I’m not using any indexes, can anyone explain why this is happening and recommend a solution?

Update 1:
I tried setting PRAGMA syncronous = OFF; and while that sped things up a little, it still get a fraction of a second slower with every INSERT SELECT to the point where it is multiple seconds for every flush after a few thousand rows. I’ll keep trying other optimizations to see if I can get to the bottom of this…

Update 2:
Clarification on what I’m doing: I’m inserting the records as they are parsed into a temporary table that’s in memory until the count reaches 400 as counted by an int in Objective-C code. Once the record count is 400, I’m doing a single INSERT SELECT to move the rows into the table on disk, then I’m doing a DELETE * from the memory table. I’m timing each part. Sqlite3 optimizes DELETE * when it has no WHERE clause so that it’s like dropping and recreating the table, and it is very quick, less than 100th of a second. It’s only the INSERT SELECT from the memory table to the disk table that is decreasing in speed each time. That query starts out taking about 0.1 seconds, and after each batch of 400 records that is inserted, the query takes about .1 to .2 seconds longer than the last one, until it’s eventually taking multiple seconds to move the 400 rows from memory to disk each time.

Update 3: Here are my table creation statements and the statement I’m using to move records from memory to disk. There are no keys at all. And yes my sqlite is configured so that the temporary table is in memory not disk.

The temporary table in memory:

1. CREATE TEMPORARY TABLE allSongsTemp (title TEXT, songId TEXT, artist TEXT, album TEXT, genre TEXT, coverArtId TEXT, path TEXT, suffix TEXT, transcodedSuffix TEXT, duration INTEGER, bitRate INTEGER, track INTEGER, year INTEGER, size INTEGER);

The table on disk:

1. CREATE TABLE allSongsUnsorted (title TEXT, songId TEXT, artist TEXT, album TEXT, genre TEXT, coverArtId TEXT, path TEXT, suffix TEXT, transcodedSuffix TEXT, duration INTEGER, bitRate INTEGER, track INTEGER, year INTEGER, size INTEGER);

The queries to flush the memory records to disk:

1. INSERT INTO allSongsUnsorted SELECT * FROM allSongsTemp;

2. DELETE * FROM allSongsTemp;

The query that is taking longer and longer each time is #3, the INSERT SELECT. The DELETE takes about 1/100th of a second each time.

Answer 1

Larry Lustig called it in the comments to the question, but doesn’t seem to be around anymore to create an answer.

Dropping and re-creating the table instead of doing DELETE * did the trick. It seems that though the delete was quick, it was causing some kind of fragmentation in memory that was slowing down each subsequent read. Unless I misread the sqlite3 docs, a DELETE * is supposed to be optimized as a DROP; CREATE; but it seems that may not be exectly the case for temporary memory tables (or maybe even filesystem tables, but I would need to test that to verify as this issue could very well only affect memory tables).