Question

I have a Sql Query which returns me over half million rows to process… The process doesn’t take really long, but I would like to speed it up a little bit with some multiprocessing. Considering the code below, is it possible to multithread something like that easily?

using (SqlDataReader reader = command.ExecuteReader())
{
    while (reader.Read())
    {
        // ...process row
    }
}

It would be perfect if I could simply get a cursor at the beginning and in the middle of the list of results. That way, I could have two thread processing the records. However the SqlDataReader doesn’t allow me to do that…

Any idea how I could achieve that?

Answer 1

Set up a producer/consumer queue, with one producer process to pull from the reader and queue records as fast as it can, but do no “processing”. Then some other number of processes (how many you want depends on your system) to dequeue and process each queued record.