Question

I have a method to process the rows from a datatable using multiple threads, it queues up all the work items and then checks that they have all been processed, not leaving the method until they have.

It seems to work fine in development, but when I put in onto the server (64-bit) for testing, it will not wait at the end of the method. It doesn’t even seem to do the Thread.Sleep() calls, as the method exits straight away.

It will continue to process the datarows after leaving the method, but that’s not what I want.

Any ideas?
Thanks

Public Sub ProcessAll(ByVal collection As DataTable, ByVal processDelegate As WaitCallback)
        Dim workItem As DataRow
        Dim availableThreads As Integer
        Dim completionPortThreads As Integer


        ThreadPool.SetMaxThreads(MAX_THREADS, MAX_THREADS)

        ' loop round processing each pending record adding them to the Thread Queue
        For Each workItem In collection.Rows
            ThreadPool.QueueUserWorkItem(processDelegate, workItem)
        Next

        ' The ThreadPool is a collection of background threads, thus we need to do something to stop the main thread from moving on
        Do
            Thread.Sleep(1000)
            ThreadPool.GetAvailableThreads(availableThreads, completionPortThreads)
            ' in the case that all threads are free (how we check all are complete) wait a few seconds just to make sure
            If availableThreads = MAX_THREADS Then
                Thread.Sleep(5000)
                ThreadPool.GetAvailableThreads(availableThreads, completionPortThreads)
            End If
        Loop While availableThreads < MAX_THREADS
End Sub

Answer 1

You shouldn’t be doing it this way to wait until all the rows have been completed.

You should instead use a notification method that will, from your processing code, tell some piece of code that “hey, I just finished a row”.

This way, you can more easily keep track of how many rows have been processed.

The way I would do it:

1. Add an Int32 value, initialized to 0, which keeps track of how many have been processed
2. Use Interlocked.Increment to increase this in your delegate
3. Set an event object in your delegate, after increasing the counter
4. In your main method, I would wait for the event to be set, and then inspect the counter. If it isn’t high enough, loop back and wait again.

ie. something like:

private volatile Int32 _Processed = 0;
private AutoResetEvent _RowProcessedEvent = new AutoResetEvent(false);

... in your delegate:
Interlocked.Increment(ref _Processed);
_RowProcessedEvent.Set();

... in your main method:
while (_Processed < collection.Rows.Count)
{
    _RowProcessedEvent.WaitOne(Timeout.Infinite);
}

don’t forget to close the event object when you’re done with it.