Question

I’m using a Producer/Consumer Pattern with a System.Collection.Concurrent.BlockingCollection<DataTable> to retrieve data from a database (producer) and create a Lucene Index on the data (consumer).

The Producer grabs 10000 records at a time and adds the set to the BlockingCollection<DataTable>. The consumer (which is a little bit slower) then grabs those 10000 and creates an index.

The blocking collection is bounded to 5 <DataTable> of 10000 rows each.

At first the program runs great but after its gets about 150000 rows in I noticed that my computers memory is maxed out and it slows to a crawl.

It seems that the BlockingCollection fails to set the underlying array slot to null after the item is taken.

Code:

    private static LuceneIndex index;
    private static BlockingCollection<DataTable> blockingCol;

    private static void Producer()
    {
        while (true)
        {
            //...get next 10000 rows
            DataTable data = GetNextSet();
            if(data.Row.Count > 0)
                blockingCol.Add(products);
            else
                break;
        }
    }

    private static void Consumer()
    {
        while (!BlockingCol.IsCompleted || BlockingCol.Count > 0)
        {
            DataTable data = blockingCol.Take();
            index.UpdateIndex(GetLuceneDocs(data));
        }
    }


 public static void Main(System.String[] args)
 {
            index = new LuceneIndex();
            blockingCol = new BlockingCollection<DataTable>(2);
            // Create the producer and consumer tasks.
            Task Prod = new Task(Producer);
            Task Con = new Task(Consumer);
            // Start the tasks.
            Con.Start();
            Prod.Start();
            // Wait for both to finish.
            try
            {
               Task.WaitAll(Con, Prod);
            }
            catch (AggregateException exc)
            {
               Console.WriteLine(exc);
            }
            finally
            {
               Con.Dispose();
               Prod.Dispose();
               blockingCol.Dispose();
             }
}

Can anyone confirm of reject this suspension? And is there any work around?

Answer 1

Yes I can confirm this. You aren’t on .NET 4.5, are you? It is supposed to be fixed there (and your comments under this answer seem to confirm this).

Anyway, write yourself a wrapper around a DataTable and clear that wrapper when you are done with the table. That makes it eligible for GC. The wrapper will not be GC’ed early but it is tiny.

class Wrapper<T> { public T Item; }