Question

I have a data logging app using sqlite3 to store different types of records that share a common header.

I put the header in one table and created separate tables for the details of each variant.

The rowid in the detail tables are the rowid of the header table. A header rowid only shows up in one of the details tables (for that variant).

I would like to fetch multiple types of records in a single query. That is, I want Sqlite to do an indexed search of the headers table to find a working set of records, and then use that set of ids to do a quick binary fetch of the variant details by rowid. So:

SELECT * FROM headers JOIN headers 
ON headers.id = variant1.id OR headers.id = variant2.id
WHERE some_header_condition

or

SELECT * FROM headers JOIN headers
ON headers.id IN (variant1.id, revariant2.id )
WHERE some_header_condition

This works, but when confronted with an OR term in the JOIN predicate, sqlite3 does a full table scan of the details tables variant1 and variant2 instead of just fetching the appropriate record by the rowid foreign key.

Something like:

0     0              2     SCAN TABLE variant2 (~5900 rows)
0     1              1     SCAN TABLE variant1 (~26588 rows)
0     2              0     SEARCH TABLE headers USING INTEGER PRIMARY KEY (rowid=?) (~2 rows)
0     0              0     EXECUTE LIST SUBQUERY 1

I can force binary searches by doing tricks like:

SELECT header.f1, variant1.f, NULL FROM header JOIN header.id = variant1.id ...
UNION ALL
SELECT header.f1, NULL, variant2.f FROM header JOIN header.id = variant2.id ...

But then the header table is accessed twice.

I could also imagine selecting the header.id(s) to a temporary table and using that to grab details using the ids IN it.

OR…I could just de-normalize the whole mess.

But all these workarounds are very inconvenient. So my question is, is there a nice JOIN query that can pick up these variants in one go with no table scans?

Answer 1

Try using outer joins:

SELECT *
FROM headers
  LEFT JOIN variant1 ON headers.id = variant1.id
  LEFT JOIN variant2 ON headers.id = variant2.id
WHERE headers...

resulting in a plan like this:

sele  order  from  deta
----  -----  ----  ----
0     0      0     SEARCH TABLE headers USING INTEGER PRIMARY KEY (rowid>? AND rowid<?) (~31250 rows)
0     1      1     SEARCH TABLE variant1 USING INTEGER PRIMARY KEY (rowid=?) (~1 rows)
0     2      2     SEARCH TABLE variant2 USING INTEGER PRIMARY KEY (rowid=?) (~1 rows)