Question

I built sqlite3 from source to include the FTS3 support and then created a new table in an existing sqlite database containing 1.5million rows of data, using

CREATE VIRTUAL TABLE data USING FTS3(codes text);

Then used

INSERT INTO data(codes) SELECT originalcodes FROM original_data;

Then queried each table with

SELECT * FROM original_data WHERE originalcodes='RH12';

This comes back instantly as I have an index on that column

The query on the FTS3 table

SELECT * FROM data WHERE codes='RH12';

Takes almost 28 seconds

Can someone help explain what I have done wrong as I expected this to be significantly quicker

Answer 1

The documentation explains:

FTS tables can be queried efficiently using SELECT statements of two different forms:

• Query by rowid. If the WHERE clause of the SELECT statement contains a sub-clause of the form “rowid = ?”, where ? is an SQL expression, FTS is able to retrieve the requested row directly using the equivalent of an SQLite INTEGER PRIMARY KEY index.
• Full-text query. If the WHERE clause of the SELECT statement contains a sub-clause of the form ” MATCH ?”, FTS is able to use the built-in full-text index to restrict the search to those documents that match the full-text query string specified as the right-hand operand of the MATCH clause.

If neither of these two query strategies can be used, all queries on FTS tables are implemented using a linear scan of the entire table.

For an efficient query, you should use

SELECT * FROM data WHERE codes MATCH 'RH12'

but this will find all records that contain the search string.

To do ‘normal’ queries efficiently, you have to keep a copy of the data in a normal table.
(If you want to save space, you can use a contentless or external content table.)