Question

I’m creating a caching system to take data from an SQLite database table using a sorted/filtered query and display it. The tables I’m pulling from can be potentially very large and, of course, I need to minimize impact on memory by only retaining a maximum number of rows in memory at any given time. This is easily done by using LIMIT and OFFSET to load only the records I need and update the cache as needed. Implementing this is trivial. The problem I’m having is determining where the insertion index is for a new record inserted into a particular query so I can update my UI appropriately. Is there an easy way to do this? So far the ideas I’ve had are:

1. Dump the entire cache, re-count the Query results (there’s no guarantee the new row will be included), refresh the cache and refresh the entire UI. I hope it’s obvious why that’s not really desirable.
2. Use my own algorithm to determine whether the new row is included in the current query, if it is included in the current cached results and at what index it should be inserted into if it’s within the current cached scope. The biggest downfall of this approach is it’s complexity and the risk that my own sorting/filtering algorithm won’t match SQLite’s.

Of course, what I want is to be able to ask SQLite: Given ‘Query A’ what is the index of ‘Row B’, without loading the entire query results. However, so far I haven’t been able to find a way to do this.

I don’t think it matters but this is all occurring on an iOS device, using the objective-c programming language.

More Info

The Query and subsequent cache is based off of user input. Essentially the user can re-sort and filter (or search) to alter the results they’re seeing. My reticence in simply recreating the cache on insertions (and edits, actually) is to provide a ‘smoother’ UI experience.

I should point out that I’m leaning toward option “2” at the moment. I played around with creating my own caching/indexing system by loading all the records in a table and performing the sort/filter in memory using my own algorithms. So much of the code needed to determine whether and/or where a particular record is in the cache is already there, so I’m slightly predisposed to use it. The danger lies in having a cache that doesn’t match the underlying query. If I include a record in the cache that the query wouldn’t return, I’ll be in trouble and probably crash.

Answer 1

The solution I came up with is not exactly simple, but it’s currently working well. I realized that the index of a record in a Query Statement is also the Count of all it’s previous records. What I needed to do was ‘convert’ all the ORDER statements in the query to a series of WHERE statements that would return only the preceding records and take a count of those records. It’s trickier than it sounds (or maybe not…it sounds tricky). The biggest issue I had was making sure the query was, in fact, sorted in a way I could predict. This meant I needed to have an order column in the Order Parameters that was based off of a column with unique values. So, whenever a user sorts on a column, I append to the statement another order parameter on a unique column (I used a “Modified Date Stamp”) to break ties.

Creating the WHERE portion of the statement requires more than just tacking on a bunch of ANDs. It’s easier to demonstrate. Say you have 3 Order columns: “LastName” ASC, “FirstName” DESC, and “Modified Stamp” ASC (the tie breaker). The WHERE statement would have to look something like this (‘?’ = record value):

WHERE
    "LastName" < ? OR
    ("LastName" = ? AND "FirstName" > ?) OR
    ("LastName" = ? AND "FirstName" = ? AND "Modified Stamp" < ?)

Each set of WHERE parameters grouped together by parenthesis are tie breakers. If, in fact, the record values of “LastName” are equal, we must then look at “FirstName”, and finally “Modified Stamp”. Obviously, this statement can get really long if you’re sorting by a bunch of order parameters.

There’s still one problem with the above solution. Mathematical operations on NULL values always return false, and yet when you sort SQLite sorts NULL values first. Therefore, in order to deal with NULL values appropriately you’ve gotta add another layer of complication. First, all mathematical equality operations, =, must be replace by IS. Second, all < operations must be nested with an OR IS NULL to include NULL values appropriately on the < operator. This turns the above operation into:

WHERE
    ("LastName" < ? OR "LastName" IS NULL) OR
    ("LastName" IS ? AND "FirstName" > ?) OR
    ("LastName" IS ? AND "FirstName" IS ? AND ("Modified Stamp" < ? OR "Modified Stamp" IS NULL))

I then take a count of the RowID using the above WHERE parameter.

It turned out easy enough for me to do mostly because I had already constructed a set of objects to represent various aspects of my SQL Statement which could be assembled to generate the statement. I can’t even imagine trying to manipulate a SQL statement like this any other way.

So far, I’ve tested using this on several iOS devices with up to 10,000 records in a table and I’ve had no noticeable performance issues. Of course, it’s designed for single record edits/insertions so I don’t really need it to be super fast/efficient.