Question

I’m having a time issue with looping through like 1 million potential rows from a database. I basically pull the rows into a DataTable and loop through them, but it is getting slow. What is the alternative out there? I can split this rows into chunks like 20,000 a piece. Can I use parallel processing in C#? Basically the code loops through every potential record that matches a certain query and tries to figure out if it is a legitimate entry. That is why every record needs to be individually visited. A record for a one object could reach 10 million rows. Approaches seem like parallel processing in multiple computers or PP in single machine with multiple cores, or some kind of data structure/approach change?

Any opinions, thoughts and guesses are helpful to make this fast and reasonable?

Answer 1

First off: Do not use DataTable for operations like these:

• it’s slow
• it’s consuming too much memory
• and you need to wait a long time before you can actually start processing the data
  • During this time additional cores are doing nothing, since reading data into a DataTable is not parralized.
  • Also while reading data the CPU is usually almost under utilized, since the network or other I/O delay is often the leading factor.

So again: Do not use DataTable for operations like these.

Instead use the DataReader. This allows you to immediately start consuming/process the data, instead of waiting for it to be loaded. The simplest version would be (sample for MS SQL Server):

var command = new SqlCommand()
{
  CommandText = "SELECT * FROM Table";
  Connection = new SqlConnection("InsertConnectionString");
};

using(var reader = command.ExecuteReader())
{
  while(reader.Read())
  {
    var values = new object[reader.FieldCount];
    reader.GetValues(values);

    // process values of row
  }
}

The reader will be blocked while executing your processing code, meaning no more rows are read from the DB.
If the processing code is heavy, it might be worth it to use the Task library to create Tasks that perform the check, which would enable you to make use of multiple cores. However, there is an overhead of creating a Task, if one Task does not contain enough ‘work’ you can batch a couple of rows together:

public void ReadData()
{
  var taskList = new List<Task<SomeResultType>>();

  var command = new SqlCommand()
  {
    CommandText = "SELECT * FROM Table";
    Connection = new SqlConnection("InsertConnectionString");
  };
  using(var reader = command.ExecuteReader())
  {
    var valueList = new List<object[]>(100);
    while(reader.Read())
    {
      var values = new object[reader.FieldCount];
      reader.GetValues(values);

      valueList.Add(values);

      if(valueList.Count == 100)
      {
        var localValueList = valueList.ToList();
        valueList.Clear();

        taskList.Add(Task<SomeResultType>.Factory.StartNew(() => Process(localValueList));
      }
    }
    if(valueList.Count > 0)
      taskList.Add(Task<SomeResultType>.Factory.StartNew(() => Process(valueList));
  }

  // this line completes when all tasks are done
  Task.WaitAll(taskList.ToArray());
}

public SomeResultType Process(List<object[]> valueList)
{
  foreach(var vals in valueList)
  {
    // put your processing code here, be sure to synchronize your actions properly
  }  
}

• The batch size (currently 100) depends on the actual processing being done and might need to be adjusted.
• Synchronizing holds it’s own challenges, you need to be very careful about shared resources