Question

Is it possible to extract the code section inside my code and have it run in multiple threads?

The app copies over data from a FoxPro db to our SQL server over a network (the files are quite huge so the bulk copy needs to happen in increments…

It works, but I’d like to bump up the speed a bit.

1) By either having the section I marked run in multiple threads, OR as an alternative,

2) not loop through each column in the datarow,

I went for the second option… (Updated code below)

CODE

private void BulkCopy(OleDbDataReader reader, string tableName, Table table)
    {
        if (Convert.ToBoolean(ConfigurationManager.AppSettings["CopyData"]))
        {
            Console.WriteLine(tableName + " BulkCopy Started.");
            try
            {
                DataTable tbl = new DataTable();
                foreach (Column col in table.Columns)
                {
                    tbl.Columns.Add(col.Name, ConvertDataTypeToType(col.DataType));
                }

                int batch = 1;
                int counter = 0;

                DataRow tblRow = tbl.NewRow();

                while (reader.Read())
                {
                    counter++;                  
 ////This section changed
                        object[] obj = tblRow.ItemArray;
                        reader.GetValues(obj);
                        tblRow.ItemArray = obj;
////**********
                    tbl.LoadDataRow(tblRow.ItemArray, true);

                    if (counter == BulkInsertIncrement)
                    {
                        Console.WriteLine(tableName + " :: Batch >> " + batch);
                        counter = PerformInsert(tableName, tbl, batch);
                        batch++;
                    }
                }

                if (counter > 0)
                {
                    Console.WriteLine(tableName + " :: Batch >> " + batch);
                    PerformInsert(tableName, tbl, counter);
                }

                tbl = null;
                Console.WriteLine("BulkCopy Success!");
            }
            catch (Exception)
            {
                Console.WriteLine("BulkCopy Fail!");
            }
            finally
            {
                reader.Close();
                reader.Dispose();
            }
            Console.WriteLine(tableName + " BulkCopy Ended.");
        }
    }

UPDATE
I went for the second option

I wasn’t aware that while inside the while(reader.Read()) loop that i could do the following. I’t helped to greatly increase the apps performance

while (reader.Read())
{
  object[] obj = tblRow.ItemArray;
  reader.GetValues(obj);
  tblRow.ItemArray = obj;
  tbl.LoadDataRow(tblRow.ItemArray, true);
}

Answer 1

This may not be the answer you’re after, but have you tried running the console application in release mode first, with just one try statement, and using indexes on the reader? It’s probably not going to increase the speed a great deal by making it multi-threaded as SQL Server will be the main bottleneck.

Of course if you don’t care too much about data integrity (for example your IDs aren’t sequential) you could change the table locking type for inserts and spin up 3-4 threads to read from certain points in the table.