So I’m writing an application that is very heavy on SQLite usage. I’m working on writing into my application an in memory caching system that will allow me to sort and filter my data (my own personal Core Data…in essence). I’m doing this because it seems to me that this is a better/faster option than to constantly make read requests from the SQLite database. Plus, most fields/columns will be searchable/sortable, and to set up indexes on each one seems less than ideal. But I’m not sure. I know the SQLite database is cached some in memory but I don’t know to what extent or how much of an advantage that would be for me. Implementing my own caching system will be complex and will probably add to my memory footprint, especially since I’m loading each table completely into memory to perform the sort/filters. I’m more than willing to do it if it helps the performance of my app, but will it? Is the SQLite caching sufficient for me to rely solely on that or will it get bogged down when the tables start getting large (10,000+ rows)? I guess I’m asking if anyone has enough experience with SQLite to recommend one over the other.
Before anyone asks: no I can’t use Core Data. Core Data isn’t flexible enough for me to use in my application.
Ok, so here’s what I’ve figured: the choice depends greatly on your requirements. I ended up removing (as much as possible) the SQLite cache, loading up what I needed, and sorting/filtering it using my own routines. This works remarkably well for me. But I’ve realized by implementing this that this wouldn’t work in a lot of situations. Specifically, I’ve done a lot to make sure that my DB size is as small as possible. I’m basically storing only simple/small text and numbers. Everything else is a reference to an outside file. This makes my database small enough to use less as a database and more as an indexing service, which works well for loading the info into memory and sorting/filtering.
So, the answer depends a lot on the database. If you’re storing large fields that might potentially take a lot of memory, it’s probably best to let SQLite handle the cache. On the other hand, if you know the fields will be small, the SQLite cache will only inflate your memory and the round trips to the database to sort/filter data will only increase your latency. Instead, it’s better to do the sorting/filtering yourself, though I will admit that’s a lot work. But in the end it made my app a lot faster than round-tripping it to the DB.