Question

I’m really struggling to iron out this issue. When I use the following code to update my database for large numbers of records it runs extremely slow. I’ve got 500,000 records to update which takes nearly an hour. During this operation, the journal file grows slowly with little change on the main SQLite db3 file – is this normal?

The operation only seems to be a problem when I have large numbers or records to update – it runs virtually instantly on smaller numbers of records.

Some other operations are performed on the database prior to this code running so could they be some how tying up the database? I’ve tried to ensure that all other connections are closed properly.

Thanks for any suggestions

using (SQLiteConnection sqLiteConnection = new SQLiteConnection("Data Source=" + _case.DatabasePath))
{
    sqLiteConnection.Open();
    using (SQLiteCommand sqLiteCommand = new SQLiteCommand("begin", sqLiteConnection))
    {
        sqLiteCommand.ExecuteNonQuery();
        sqLiteCommand.CommandText = "UPDATE CaseFiles SET areaPk = @areaPk, KnownareaPk = @knownareaPk WHERE mhash = @mhash";
        var pcatpk = sqLiteCommand.CreateParameter();
        var pknowncatpk = sqLiteCommand.CreateParameter();
        var pmhash = sqLiteCommand.CreateParameter();
        pcatpk.ParameterName = "@areaPk";
        pknowncatpk.ParameterName = "@knownareaPk";
        pmhash.ParameterName = "@mhash";
        sqLiteCommand.Parameters.Add(pcatpk);
        sqLiteCommand.Parameters.Add(pknowncatpk);
        sqLiteCommand.Parameters.Add(pmhash);
        foreach (CatItem CatItem in _knownFiless)
        {

            if (CatItem.FromMasterHashes == true)
            {
                pcatpk.Value = CatItem.areaPk;
                pknowncatpk.Value = CatItem.areaPk;
                pmhash.Value = CatItem.mhash; 
            }
            else
            {
                pcatpk.Value = CatItem.areaPk;
                pknowncatpk.Value = null;
                pmhash.Value = CatItem.mhash; 
            }
            sqLiteCommand.ExecuteNonQuery();
        }
        sqLiteCommand.CommandText = "end";
        sqLiteCommand.ExecuteNonQuery();
        sqLiteCommand.Dispose();
        sqLiteConnection.Close();
    }
    sqLiteConnection.Close();
}

Answer 1

The first thing to ensure that you have an index on mhash.
Group commands into batches.
Use more than one thread.

Or [inserted]

Bulk import the records to a temporary table. Create an index on the mhash column. Perform a single update statement to update the records.