Question

I have an ASP.NET app, very basic, but right now too much code to post if we’re lucky and I don’t have to.

We have a class called ReportGenerator. On a button click, method GenerateReports is called. It makes an async call to InternalGenerateReports using ThreadPool.QueueUserWorkItem and returns, ending the ASP.NET response. It doesn’t provide any completion callback or anything.

InternalGenerateReports creates and maintains five threads in the threadpool, one report per thread, also using QueueUserWorkItem, by ‘creating’ five threads, also with and waiting until calls on all of them complete, in a loop. Each thread uses an ASP.NET ReportViewer control to render a report to HTML. That is, for 200 reports, InternalGenerateReports should create 5 threads 40 times. As threads complete, report data is queued, and when all five have completed, report data is flushed to disk.

My biggest problems are that after running for just one report, the aspnet process is ‘hung’, and also that at around 200 reports, the app just hangs.

I just simplified this code to run in a single thread, and this works fine. Before we get into details like my code, is there anything obvious in the above scendario that might be wrong?

Here is an abreviated example of the code:

public class SscceReports
{
    Dictionary<Guid, AsyncReportCreator> runningWorkers = new Dictionary<Guid, AsyncReportCreator>();
    public void GenerateReports(Queue<int> reportKeys)
    {
        int goodPoolSize = System.Environment.ProcessorCount;
        System.Threading.ThreadPool.SetMaxThreads(goodPoolSize + 10, goodPoolSize * 10);
        System.Threading.ThreadPool.QueueUserWorkItem(new System.Threading.WaitCallback(InternalGenerateReports), reportKeys);
    }

    void InternalGenerateReports(object state)
    {
        Queue<int> reportKeys = state as Queue<int>;
        while (reportKeys.Count > 0)
        {
            for (int i = 0; i < 5 && reportKeys.Count > 0; i++)
            {
                Guid workerId = Guid.NewGuid();
                int rk = (int) reportKeys.Dequeue();
                AsyncReportCreator asrc = new AsyncReportCreator(rk);
                runningWorkers.Add(workerId, asrc);
                asrc.WorkComplete += CompleteCallBack;
                System.Threading.ThreadPool.QueueUserWorkItem(asrc.StartWork);
            }
            while (runningWorkers.Count > 0)
                System.Threading.Thread.Sleep(500);
        }
        while (runningWorkers.Count > 0)
            System.Threading.Thread.Sleep(5000);
    }

    void CompleteCallBack(object state)
    {
        // Write queued report content to disk.
        runningWorkers.Remove((Guid) state);
    }
}

public class AsyncReportCreator
{
    public event System.Threading.WaitCallback WorkComplete;
    private int key;
    public AsyncReportCreator(int reportKey)
    {
        key = reportKey;
    }

    public void StartWork(object state)
    {
        // Create report;
        WorkComplete(state);
    }
}

Answer 1

The .NET ThreadPool by default has 25 threads per processor, so if you’re generating 200 reports when you call InternalGenerateReports it will queue 1,000 work items in the ThreadPool (200 reports * 5 work items for each InternalGenerateReports). Only 25 will be active at a time, but you can see that this model is probably not suitable for you.

The thread pool has a default limit of
25 threads per available processor,
which could be changed using
CorSetMaxThreads as defined in the
mscoree.h file. Each thread uses the
default stack size and runs at the
default priority. Each process can
have only one operating system thread
pool.

Think about using a producer/consumer pattern and instead of queuing 25 work items (i.e 25 threads), just create a few consumer threads (matching the number of processors/cores you have) which process requests for report generation from a central blocking queue which is populated by the producer(s). That should make things a bit more reasonable…

There are some articles that say you shouldn’t use a ThreadPool with ASP.NET:

You can use the ThreadPool in
exactly the same way in ASP.NET and it
works just as you would expect. The
problem is not in the ThreadPool
itself but in what else ASP.NET uses
it for at the same time. ASP.NET is
multi-threaded by design and it uses
the ThreadPool to serve pages and
content mapped to the ASP.NET ISAPI
filter.

If you also use the ThreadPool, then
ASP.NET has fewer threads to utilize
and requests are put on hold until the
pool returns a free thread. This might
not be a problem for a low traffic
site, but more popular sites can get
into trouble. Low traffic sites can
get into trouble if they use the
ThreadPool a lot.

Update

I think I found your issue… you’re queuing asrc.StartWork but you’re not passing in a state so when CompleteCallBack is called it doesn’t have anything to remove from the runningWorkers. Here is an improved version for you:

public class CountDownLatch {
    private volatile int m_remain;
    private EventWaitHandle m_event;

    public CountDownLatch(int count) {
        m_remain = count;
        m_event = new ManualResetEvent(false);
    }

    public void Signal() {
        // The last thread to signal also sets the event.
        if (Interlocked.Decrement(ref m_remain) == 0)
            m_event.Set();
    }

    public void Wait() {
        m_event.WaitOne();
    }
}

public class SscceReports
{
    public void GenerateReports(Queue<int> reportKeys)
    {
        int goodPoolSize = System.Environment.ProcessorCount;
        System.Threading.ThreadPool.SetMaxThreads(goodPoolSize + 10, goodPoolSize * 10);
        System.Threading.ThreadPool.QueueUserWorkItem(o=>
        {
            InternalGenerateReports(reportKeys);
        });
    }

    void InternalGenerateReports(Queue<int> reportKeys)
    {
        // assuming that the reportKeys.Count contains only 5 keys,
        // we create a countdown latch to hande the same number of keys
        CountDownLatch latch = new CountDownLatch(reportKeys.Count);
        foreach( int rk in reportKeys)
        {
            AsyncReportCreator asrc = new AsyncReportCreator(rk);
            asrc.WorkComplete += CompleteCallBack;
            System.Threading.ThreadPool.QueueUserWorkItem(o=>
            {
                asrc.StartWork(latch);
            });
        }

        // Wait for all the tasks to complete instead of sleeping
        latch.Wait(); // <- blocks untill all (5) tasks call latch.Signal()
    }

    void CompleteCallBack(CountDownLatch latch)
    {
        // Write queued report content to disk.
        latch.Signal(); 
    }
}

public class AsyncReportCreator
{
    public delegate void WorkComplete(CountDownLatch latch);
    public AsyncReportCreator(int reportKey)
    {
        key = reportKey;
    }

    public void StartWork(CountDownLatch latch)
    {
        // Create report;
        WorkComplete(latch);
    }
}