Question

I have a database, which has a table with fields like “title, album, artist…” and it also has many fields with html content for every record (up to 30).

Problem is, that this database has tens of thousands of records and is hundreds of megabytes large because of the html content. Because of the size of the sqlite file the search is very slow (also inserting new elements in a transaction is very slow ~10-30 second for 200 new rows). The very first LIKE query can take 10-15 seconds, other searches are fast enough (indices are created and work ok). When I removed the html content from the database the search was always instant.

So the question is, what is the best way to store that additional html content? Right now I play with the option to store it in separate files, but it can generate up to 600k files and more in the future, which is quiet slow to create. Storing the files in a zip archive will probably hit its file number limit. Other options are to zip files per table row, store the html in a separate table in the same database, or to create a separate database file for the html content.

What will give me the best performance? Or are there other better options? I need quick insert, update and serach.

Answer 1

After some days of experimenting I came to this conclusion:

• one database file with one table was the slowest (up to 10 seconds)
• one database with two tables was twice as fast in the worst case scenario as one table
• fastest was to have two separate database files. one with data needed for search and the other for the huge html data. this is almost instant in the worst case ~300ms and in normal usage it is instant

So I reccommend to use two separate database files in this scenario. If someone does not come with a faster/better solution I will accept this as the answer.