Question

I am currently writing some software in visual studio to analyse large amounts of data from an Access database using SQL. I have code to make a new calculated variable but am struggling with the amount of time it takes to write the data back into Access.

I am currently using some vb com code to communicate with my Access Database which is running in 2002/3 comparability mode. The following is my current code which runs a function in a loop to write to the database.

cnnOLEDB = New OleDbConnection
    cnnOLEDB.ConnectionString = "Provider=Microsoft.Jet.OLEDB.4.0;Data Source=" & DataDirectoryName & DatabaseFileName
    cnnOLEDB.Open()

    'cmdOLEDB = New OleDbCommand
    cmdOLEDB.Connection = cnnOLEDB

    ColumnString = "ID_VAR, ID_PAR, TimeValue, strValue, ID_UPL"
    For RecordCounter = 0 To CalcData.GetLength(1) - 1
        Var_ID = Var_ID + 1
        ValueString = Format(Var_ID, "0") & ", " & Format(Parameter, "0") & ", #" & Date2String(CDate(CalcData(0, RecordCounter))) & "#, " & CalcData(CalcData.GetLength(0) - 1, RecordCounter) & ", " & Format(AsUpload, "0")
        If DatabaseConnectionInsert("INSERT INTO " & TableName & " (" & ColumnString & ") VALUES (" & ValueString & ")", "Non-Query") = "Error" Then GoTo Close
    Next

    cnnOLEDB.Close()

Here is the Function:

Public Function DatabaseConnectioninsert(ByVal Query As String, ByVal Task As String) As String
        'On Error GoTo Err

        'If cnnOLEDB.State = ConnectionState.Open Then cnnOLEDB.Close()
        cmdOLEDB.CommandText = Query

        Select Case Task
            Case "Read Recordset"
                rdrOLEDB = cmdOLEDB.ExecuteReader()
                DatabaseConnectioninsert = "Read Recordset"
            Case "Read Scalar"
                DatabaseConnectioninsert = cmdOLEDB.ExecuteScalar
            Case "Non-Query"
                cmdOLEDB.ExecuteNonQuery()
                DatabaseConnectioninsert = "Non-Query"
        End Select

        Exit Function
Err:
        MsgBox("Database connection error.")
        DatabaseConnectioninsert = "Error"


    End Function

I am currently trying to insert ~4500 records into the Access Database for each Parameter which takes ~3minutes. However when the project goes live it will have to deal with over 100000 records per Parameter so it is no where near fast enough.

To solve this issue I am thinking of either updating my code to .net or creating a record set, so I can move all of the data in Access at once. Can anyone give me some advice as to which will have the greatest impact to improving the speed of the inserts. I am running visual studio 2005 and Access 2007, updating the database to 2007 rather than compatibility mode is possible but not ideal , however my current code can’t access it.

Thank you for your help

Josh

Answer 1

As ridiculous as it sounds, the very best performance you will get on an Access database is using the ancient DAO COM library. Use a RecordSet object to add the records one at a time in a loop and reference the fields by their index (ordinal position) rather than their names. You will find it much, much quicker than using oleDB.ExecuteNonQuery.

See the solution given here for more information. It’s C# but it’s easy enough to follow and convert to VB.NET if you want to try it out.

Edit
In deference to Remou’s comments below: it would appear that Microsoft have in fact been keeping DAO technology up to date – in spite of declaring it obsolete back in 2002 – but you have to use the Office Access Redistributable rather than the better known DAO 3.6 library.