Question

I am working on improving database synchronization application that requires to work with millions of records at the time. I will employ SqlBulkCopy operation and then will use MERGE on the database to merge temporary tables with current tables. But before that I create couple tables that I will add to the database. When I use regular loops, it works fine, but when I create Parallel.ForEach loops, it processes around 100 records and then just hangs. No errors, no nothing… I assume it is hitting some sort of threshold, but don’t know how to handle it. When I select 2 parallel threads, it works fine, but it is getting stuck with > 2 threads without any errors.

DataTable PRODUCT = new DataTable();

ParallelOptions parOptions = new ParallelOptions();
parOptions.MaxDegreeOfParallelism = 5; //use max (5) threads that are allowed.
Parallel.ForEach(dtPs.AsEnumerable(), parOptions, dtPs_row =>
{
   try
   { 
      //some declarations
      DataRow newProductRow = PRODUCT.NewRow();
      newProductRow.SetField("ID", mId);
      newProductRow.SetField("NAME", name);
      PRODUCT.Rows.Add(newProductRow);
   }
}

Answer 1

Here is an example with your code:

    DataTable PRODUCT = new DataTable();
    Object lockobj = new Object();


    ParallelOptions parOptions = new ParallelOptions();
    parOptions.MaxDegreeOfParallelism = 5; //use max (5) threads that are allowed.
    Parallel.ForEach(dtPs.AsEnumerable(), parOptions, dtPs_row =>
    {
        try
        { 
            //some declarations
            lock(lockobj)
            {
                DataRow newProductRow = PRODUCT.NewRow();
                newProductRow.SetField("ID", mId);
                newProductRow.SetField("NAME", name);
                PRODUCT.Rows.Add(newProductRow);
            }
        }
    }

Assuming that Object lockobj = new Object(); is outside the method at the class level.