Question

I’m using ADO to save data to an MS Access database. It was taking quite a while to save the data to file (about 7 seconds – which is too long for our purposes). I looked at the number of SQL queries being run, and it’s about 4200; although there isn’t a whole heap of data.

The database connection seems to be the bottleneck. Do you know of any way to decrease the amount of time this takes; either by somehow combining multiple statements into one to reduce overhead, or some ADO/MS-Access trick?

Can you, for instance, insert multiple rows into a table at once, and would this be noticeably faster?

Extra info:

One reason we have so many queries is that we insert a row, and then have another query to retrieve its autoincremented ID; then use this ID to insert several more rows, linking them to the first

In response to several comments and responses: I am leaving the connection open the entire time, and performing it as a single transaction with BeginTransaction() and CommitTransaciton()

Answer 1

Some folk have posted that @@IDENTITY would be fast, so here’s a proof (using VBA) of how my INSERT INTO two tables at once via a VIEW trick is about three times faster than doing two INSERTS and grabbing the @@IDENTITY values each time… which is hardly surprising because the latter involves three Execute statements and the former only involves one 🙂

On my machine for the 4200 iterations, the VIEW trick took 45 seconds and the @@IDENTITY approach took 127 seconds:

Sub InitInerts()
  On Error Resume Next
  Kill Environ$("temp") & "\DropMe.mdb"
  On Error GoTo 0
  Dim cat
  Set cat = CreateObject("ADOX.Catalog")
  With cat
    .Create _
        "Provider=Microsoft.Jet.OLEDB.4.0;" & _
        "Data Source=" & _
        Environ$("temp") & "\DropMe.mdb"

    With .ActiveConnection

      Dim Sql As String

      Sql = _
      "CREATE TABLE TableA" & vbCr & "(" & vbCr & "   ID IDENTITY NOT" & _
      " NULL UNIQUE, " & vbCr & "   a_col INTEGER NOT NULL" & vbCr & ")"
      .Execute Sql

      Sql = _
      "CREATE TABLE TableB" & vbCr & "(" & vbCr & "   ID INTEGER NOT" & _
      " NULL UNIQUE" & vbCr & "      REFERENCES TableA (ID)," & _
      "  " & vbCr & "   b_col INTEGER NOT NULL" & vbCr & ")"
      .Execute Sql

      Sql = _
      "CREATE VIEW TestAB" & vbCr & "(" & vbCr & "   a_ID, a_col, " & vbCr & " " & _
      "  b_ID, b_col" & vbCr & ")" & vbCr & "AS " & vbCr & "SELECT A1.ID, A1.a_col," & _
      " " & vbCr & "       B1.ID, B1.b_col" & vbCr & "  FROM TableA AS" & _
      " A1" & vbCr & "       INNER JOIN TableB AS B1" & vbCr & "    " & _
      "      ON A1.ID = B1.ID"
      .Execute Sql

    End With
    Set .ActiveConnection = Nothing
  End With
End Sub

Sub TestInerts_VIEW()

  Dim con
  Set con = CreateObject("ADODB.Connection")
  With con
    .Open _
        "Provider=Microsoft.Jet.OLEDB.4.0;" & _
        "Data Source=" & _
        Environ$("temp") & "\DropMe.mdb"

    Dim timer As CPerformanceTimer
    Set timer = New CPerformanceTimer
    timer.StartTimer

    Dim counter As Long
    For counter = 1 To 4200
      .Execute "INSERT INTO TestAB (a_col, b_col) VALUES (" & _
                   CStr(counter) & ", " & _
                   CStr(counter) & ");"
    Next

    Debug.Print "VIEW = " & timer.GetTimeSeconds

  End With

End Sub

Sub TestInerts_IDENTITY()

  Dim con
  Set con = CreateObject("ADODB.Connection")
  With con
    .Open _
        "Provider=Microsoft.Jet.OLEDB.4.0;" & _
        "Data Source=" & _
        Environ$("temp") & "\DropMe.mdb"

    Dim timer As CPerformanceTimer
    Set timer = New CPerformanceTimer
    timer.StartTimer

    Dim counter As Long
    For counter = 1 To 4200
      .Execute "INSERT INTO TableA (a_col) VALUES (" & _
          CStr(counter) & ");"

      Dim identity As Long
      identity = .Execute("SELECT @@IDENTITY;")(0)

      .Execute "INSERT INTO TableB (ID, b_col) VALUES (" & _
                   CStr(identity) & ", " & _
                   CStr(counter) & ");"

    Next

    Debug.Print "@@IDENTITY = " & timer.GetTimeSeconds

  End With

End Sub

What this shows is the the bottleneck now is the overhead associated with executing multiple statements. What if we could do it in just one statement? Well, guess what, using my contrived example, we can. First, create a Sequence table of unique integers, being a standard SQL trick (every database should have one, IMO):

Sub InitSequence()

  Dim con
  Set con = CreateObject("ADODB.Connection")
  With con
    .Open _
        "Provider=Microsoft.Jet.OLEDB.4.0;" & _
        "Data Source=" & _
        Environ$("temp") & "\DropMe.mdb"

    Dim sql As String

    sql = _
        "CREATE TABLE [Sequence]" & vbCr & "(" & vbCr & "   seq INTEGER NOT NULL" & _
        " UNIQUE" & vbCr & ");"
    .Execute sql

    sql = _
        "INSERT INTO [Sequence] (seq) VALUES (-1);"
    .Execute sql

    sql = _
        "INSERT INTO [Sequence] (seq) SELECT Units.nbr + Tens.nbr" & _
        " + Hundreds.nbr + Thousands.nbr AS seq FROM ( SELECT" & _
        " nbr FROM ( SELECT 0 AS nbr FROM [Sequence] UNION" & _
        " ALL SELECT 1 FROM [Sequence] UNION ALL SELECT 2 FROM" & _
        " [Sequence] UNION ALL SELECT 3 FROM [Sequence] UNION" & _
        " ALL SELECT 4 FROM [Sequence] UNION ALL SELECT 5 FROM" & _
        " [Sequence] UNION ALL SELECT 6 FROM [Sequence] UNION" & _
        " ALL SELECT 7 FROM [Sequence] UNION ALL SELECT 8 FROM" & _
        " [Sequence] UNION ALL SELECT 9 FROM [Sequence] ) AS" & _
        " Digits ) AS Units, ( SELECT nbr * 10 AS nbr FROM" & _
        " ( SELECT 0 AS nbr FROM [Sequence] UNION ALL SELECT" & _
        " 1 FROM [Sequence] UNION ALL SELECT 2 FROM [Sequence]" & _
        " UNION ALL SELECT 3 FROM [Sequence] UNION ALL SELECT" & _
        " 4 FROM [Sequence] UNION ALL SELECT 5 FROM [Sequence]" & _
        " UNION ALL SELECT 6 FROM [Sequence] UNION ALL SELECT" & _
        " 7 FROM [Sequence] UNION ALL SELECT 8 FROM [Sequence]" & _
        " UNION ALL SELECT 9 FROM [Sequence] ) AS Digits )" & _
        " AS Tens, ( SELECT nbr * 100 AS nbr FROM ( SELECT" & _
        " 0 AS nbr FROM [Sequence] UNION ALL SELECT 1 FROM" & _
        " [Sequence] UNION ALL SELECT 2 FROM [Sequence] UNION"
    sql = sql & _
        " ALL SELECT 3 FROM [Sequence] UNION ALL SELECT 4 FROM" & _
        " [Sequence] UNION ALL SELECT 5 FROM [Sequence] UNION" & _
        " ALL SELECT 6 FROM [Sequence] UNION ALL SELECT 7 FROM" & _
        " [Sequence] UNION ALL SELECT 8 FROM [Sequence] UNION" & _
        " ALL SELECT 9 FROM [Sequence] ) AS Digits ) AS Hundreds," & _
        " ( SELECT nbr * 1000 AS nbr FROM ( SELECT 0 AS nbr" & _
        " FROM [Sequence] UNION ALL SELECT 1 FROM [Sequence]" & _
        " UNION ALL SELECT 2 FROM [Sequence] UNION ALL SELECT" & _
        " 3 FROM [Sequence] UNION ALL SELECT 4 FROM [Sequence]" & _
        " UNION ALL SELECT 5 FROM [Sequence] UNION ALL SELECT" & _
        " 6 FROM [Sequence] UNION ALL SELECT 7 FROM [Sequence]" & _
        " UNION ALL SELECT 8 FROM [Sequence] UNION ALL SELECT" & _
        " 9 FROM [Sequence] ) AS Digits ) AS Thousands;"
    .Execute sql

  End With

End Sub

Then use the Sequence table to enumerate the values from 1 to 42000 and construct rows in a single INSERT INTO..SELECT statement:

Sub TestInerts_Sequence()

  Dim con
  Set con = CreateObject("ADODB.Connection")
  With con
    .Open _
        "Provider=Microsoft.Jet.OLEDB.4.0;" & _
        "Data Source=" & _
        Environ$("temp") & "\DropMe.mdb"

    Dim timer As CPerformanceTimer
    Set timer = New CPerformanceTimer
    timer.StartTimer

    .Execute "INSERT INTO TestAB (a_col, b_col) " & _
             "SELECT seq, seq " & _
             "FROM Sequence " & _
             "WHERE seq BETWEEN 1 AND 4200;"

    Debug.Print "Sequence = " & timer.GetTimeSeconds



  End With

End Sub

That executes on my machine in 0.2 of a second!