Question

If I have a ConcurrentQueue, is there a preferred way to consume it with a Linq statement? It doesn’t have a method to dequeue all the items as a sequence, and it’s enumerator doesn’t remove items.

I’m doing batch consumption, meaning periodically I want to process the queue and empty it, instead of processing it until it is empty and blocking until more items are enqueued. BlockingCollection doesn’t seem like it will work because it will block when it gets to the last item, and I want that thread to do other stuff, like clear other queues.

static ConcurrentQueue<int> MyQueue = new ConcurrentQueue<int>();
void Main()
{
    MyQueue.Enqueue(1);MyQueue.Enqueue(2);MyQueue.Enqueue(3);MyQueue.Enqueue(4);MyQueue.Enqueue(5);

    var lst = MyQueue.ToLookup(x => x.SomeProperty);
    //queue still has all elements
    MyQueue.Dump("queue");  
}

For now, I’ve made a helper method

static IEnumerable<T> ReadAndEmptyQueue<T>(this ConcurrentQueue<T> q)
{
    T item;
    while(q.TryDequeue(out item))
    {
        yield return item;
    }
}

var lk = MyQueue.ReadAndEmptyQueue().ToLookup(x => x.SomeProperty);
MyQueue.Dump(); //size is now zero

Is there a better way, or am I doing it right?

Answer 1

Your approach is very reasonable, in my opinion. Allowing a consumer to empty the queue in this fashion is clean and simple.

BlockingCollection doesn’t seem like it will work because it will block when it gets to the last item, and I want that thread to do other stuff, like clear other queues.

The one thing I’d mention – sometimes, from a design standpoint, it’s easier to just fire off a separate consumer thread per queue. If you do that, each BlockingCollection<T> can just use GetConsumingEnumerable() and block as needed, as they’ll be in a wait state when the queue is empty.

This is the approach I take more frequently, as it’s often much simpler from a synchronization standpoint if each collection has one or more dedicated consumers, instead of a consumer switching between what it’s consuming.