Question

i have a local SQLite database

TABLE DETAILS

-- Describe PREFIX_LIST
CREATE TABLE PREFIX_LIST(ITEM VARCHAR(25) PRIMARY KEY)

-- Describe SUFFIX_LIST
CREATE TABLE SUFFIX_LIST(ITEM VARCHAR(25) PRIMARY KEY)

-- Describe VALID_LIST
CREATE TABLE VALID_LIST (
    "PART1" TEXT,
    "PART2" TEXT,
    PRIMARY KEY(PART1, PART2)
)

now this list is really huge, and i need need to save data from it.

Here is my implementation.

SQLiteConnection con = null;
SQLiteCommand cmd = null;
Connect(DbPath, ref con, ref cmd);

cmd.CommandText =
    "SELECT PART1 || '@' || PART2 FROM VALID_LIST 
 WHERE NOT EXISTS 
   (SELECT * FROM PREFIX_LIST WHERE VALID_LIST.PART1 LIKE '%' || ITEM || '%') 
   AND NOT EXISTS
   (SELECT * FROM SUFFIX_LIST WHERE VALID_LIST.PART2 LIKE '%' || ITEM || '%')";

var reader = cmd.ExecuteReader();

if (reader.HasRows)
{
    string savePath;

    if (SaveTextFile(out savePath) == DialogResult.OK)
    {
        TextWriter writer = new StreamWriter(savePath);
        while (reader.Read())
        {
            writer.WriteLine(reader.GetString(0));
        }
        writer.Close();
        writer.Dispose();
    }

}

reader.Close();
reader.Dispose();
cmd.Dispose();
con.Close();
con.Dispose();

MessageBox.Show("List Saved!.", Application.ProductName, MessageBoxButtons.OK, MessageBoxIcon.Information);

I need some better way i can save list faster.
total entries in VALID_LIST is 2639117

and it took 15 minutes to save it for the above SQL QUERY!

please lmk if the sql query can be optimized!

Thanks in advance

Answer 1

Queries with LIKE are going to be very slow in general unless the wildcard is attached to the suffix. A predicate such as LIKE '%foo' cannot be indexed via typical string indexing.

You can however replace heavy LIKE usage in sqlite with its full text search (FTS) feature.

The FTS3 and FTS4 extension modules allows users to create special
tables with a built-in full-text index (hereafter “FTS tables”). The
full-text index allows the user to efficiently query the database for
all rows that contain one or more words (hereafter “tokens”), even if
the table contains many large documents.

They have an example that look promising in terms of performance on your use case.

CREATE VIRTUAL TABLE enrondata1 USING fts3(content TEXT);     /* FTS3 table */
CREATE TABLE enrondata2(content TEXT);                        /* Ordinary table *

SELECT count(*) FROM enrondata1 WHERE content MATCH 'linux';  /* 0.03 seconds */
SELECT count(*) FROM enrondata2 WHERE content LIKE '%linux%'; /* 22.5 seconds */