Question

I have a .DBF file (Well, 6 DBF files with the same structure) that have over a million and a half rows. I also have a C# app that is migrating this data into a SQL database using an API we made.

The program starts quickly, being able to process 30 or 40 rows a second, but it gradually slows down over time, and I don’t know why. I believe I am disposing of objects at a good pace.

The connection string I am using in the vfpoledb is

"Provider=vfpoledb;Data Source=" + sourceDBFolder + ";Collating Sequence=machine;MVCOUNT=32000;ENGINEBEHAVIOR=90;TABLEVALIDATE=0;REFRESH=0";

Where sourceDBFolder is a path on disk. I also execute the following code before starting the task:

System.Data.OleDb.OleDbCommand oRefreshCommand = oConn.CreateCommand();
oRefreshCommand.CommandText = "EXECSCRIPT([SET REFRESH TO 0,0])";
oRefreshCommand.ExecuteNonQuery();

Here is the relevant code. The where clauses in the select statement will cause a bottleneck if they are not put in the order below.

string[] noteTables = new string[] { "note1", "note2", "note3", "note4", "note5", "note6" };
foreach (long lNoteKey in oCaseLookupTable.Keys) {
  for (int y = 0; y <= 5; y++) {
    System.Data.OleDb.OleDbCommand oNotesCmd = oConn.CreateCommand();
    oNotesCmd.CommandText = "SELECT NOTEDATE, NOTEDESC, ENTEREDBY FROM " + noteTables[y] + " WHERE NOTEPOINT = " + lNoteKey.ToString() + " AND NOTEDESC NOT LIKE 'Folder accessed%'";
    DataTable oNotesTable = new DataTable();
    oNotesTable.Load(oNotesCmd.ExecuteReader());
    foreach (DataRow oRow in oNotesTable.Rows) { 
      //Do processing on rows, Note is my created class.
      Note oNote = new Note();
      oNote.NoteValue = oRow["NOTEDESC"].ToString().Trim();
      oNote.ReferenceID = oCaseLookupTable[lNoteKey];
      DateTime createdDate;
      if (DateTime.TryParse(oRow["NOTEDATE"].ToString().Trim(), out createdDate))
        oNote.CreatedDate = createdDate;
      else
        oNote.CreatedDate = DateTime.Now;
      Result oNoteResult = oNote.Insert();
    }
    oNotesTable.Dispose();
    oNotesCmd.Dispose();
  }
}

Simply put, I don’t understand why this is gets gradually slower and slower. PerfMon doesn’t show any blocks of managed memory growing over time. I try to keep my DataTable small by making continuous calls to the DBF file. In general, the largest number of rows that will be returned from query is 1,000.

Answer 1

The issue was some database triggers on tables on the SQL side. Disabling them for the duration of the operation (They’re not needed in this case) solved the issue.