Question

On my system, ~86000 SQLite insertions took up to 20 minutes, means ~70 insertions per second. I have to do millions, how can I speed up it? Calling Open() and Close() on SQLiteConnection object for every line can slow down performance? Can transactions help?

Typical insertion method for a single line:

public int InsertResultItem(string runTag, int topicId,
    string documentNumber, int rank, double score)
{
    // Apre la connessione e imposta il comando
    connection.Open();

    command.CommandText = "INSERT OR IGNORE INTO Result "
    + "(RunTag, TopicId, DocumentNumber, Rank, Score) " +
    "VALUES (@RunTag, @TopicId, @DocumentNumber, @Rank, @Score)";

    // Imposta i parametri
    command.Parameters.AddWithValue("@RunTag", runTag);
    command.Parameters.AddWithValue("@TopicId", topicId);
    command.Parameters.AddWithValue("@DocumentNumber", documentNumber);
    command.Parameters.AddWithValue("@Rank", rank);
    command.Parameters.AddWithValue("@Score", score);

    // Ottieni il risultato e chiudi la connessione
    int retval = command.ExecuteNonQuery();
    connection.Close();

    return retval;
}

As you can see, insertions are very simple ones.

Answer 1

You definitely need a transaction. If you don’t, SQLite starts its own transaction for every insert command so you’re effectively doing 86000 transactions as is.

It looks you’re also opening and closing the connection each time, along with resetting the CommandText each time. This is unnecessary and doubtless slowing you down, it’ll go much faster if you:

• Open the connection once
• Build the command once , adding the parameters to it once.
• Start the transaction
• Loop through, changing the parameter values only before calling ExecuteNonQuery
• Commit the transaction.
• Close the connection.

I think you could reduce your 20 minutes down to just a few seconds this way.

Edit: this is what I mean:

public void InsertItems()
{
    SQLiteConnection connection  = new SQLiteConnection(SomeConnectionString);
    SQLiteCommand command = connection.CreateCommand();
    SQLiteTransaction transaction = connection.BeginTransaction();

    command.CommandText = "INSERT OR IGNORE INTO Result "
+ "(RunTag, TopicId, DocumentNumber, Rank, Score) " +
  "VALUES (@RunTag, @TopicId, @DocumentNumber, @Rank, @Score)";

    command.Parameters.AddWithValue("@RunTag", "");
    command.Parameters.AddWithValue("@TopicId", "");
    command.Parameters.AddWithValue("@DocumentNumber", "");
    command.Parameters.AddWithValue("@Rank", "");
    command.Parameters.AddWithValue("@Score", "");

    foreach ( /* item to loop through and add to db */ )
    {
        InsertResultItem(runTag, topicId, documentNumber, rank, score, command);
    }

    transaction.Commit();
    command.Dispose();
    connection.Dispose();
}

public int InsertResultItem(string runTag, int topicId, string documentNumber, int rank, double score, SQLiteCommand command)
{
    command.Parameters["@RunTag"].Value = runTag;
    command.Parameters["@TopicId"].Value = topicId;
    command.Parameters["@DocumentNumber"].Value = documentNumber;
    command.Parameters["@Rank"].Value = rank;
    command.Parameters["@Score"].Value = score;
    return command.ExecuteNonQuery();
}

It only uses one connection, one transaction and one command, so all you’re changing is the parameter values each time.