Question

I’m using sqlite for a small validation application. I have a simple one table database with 4 varhchar columns and one integer primary key. There are close to 1 million rows in the table. I have optimised it and done a vacuum on it.

I am using the following query to retrieve a presence count from the table. I have changed the fields and names for privacy.

                  SELECT 
                  count(*) as 'test'
                  FROM
                  my_table
                  WHERE
                  LOWER(surname) = LOWER('Oliver')
                  AND
                  UPPER(address_line2) = UPPER('Somewhere over the rainbow')  
                  AND
                  house_number IN ('3','4','5');

This query takes about 1.5-1.9 seconds to run. I have tried indexes and they make no difference really. This time may not sound bad but I have to run this test about 40,000 times on a read in csv file so as you may imagine it adds up pretty quickly. Any ideas on how to reduce the execution time. I normally develop in mssql or mysql so if there are some tricks I am missing in sqlite I would be happy to hear them.

All the best.

Answer 1

When you use a function over an indexed column, SQLite cannot use the index, because the function may not preserve the ordering — i.e. there can be functions such as 1>2, but F(1)<F(2). There are some ways to solve this situation, though:

1. If you want to use indexes to make your query faster, you must save
   the value in a fixed case (upper or lower) and then convert only the
   query parameter to the same case:

SELECT count(*) as 'test'
FROM my_table
WHERE surname = LOWER('Oliver')

1. You can use the case-insensitive LIKE operator (I don’t know how indexes are affected!):

SELECT count(*) as 'test'
FROM my_table
WHERE surname LIKE 'Oliver';

1. Or you can create each column as text collate nocase and don’t worry about case differences regarding this column anymore:

CREATE TABLE my_table (surname text collate nocase, <... other fields here ...>);
SELECT count(*) as 'test'
FROM my_table
WHERE surname ='Oliver';

You can find more information about the = and LIKE operators here.