Question

I have a simple while loop

IEnumerable<Foo> collection;
while (!bc.IsCompleted)
{
   collection = bc.Take();
}

bc is a BlockingCollection<IEnumerable<Foo>>. The bc contains 9 IEnumerable collections and a total of 2.6million Foo objects. The loop takes about 640ms to run on my machine. As soon as I add a foreach loop after Take() within the while loop the time it takes to run explodes to 2400ms.

foreach(Foo foo in collection)
{
}

Iterating over 2.6 million elements within a List or Foo[] or IEnumerable that I setup separately, took about 54ms.

The same happens if instead of the foreach loop I simply add a collection conversion such as

List<Foo> fooList = collection.ToList();

or

Foo[] fooArray = collection.ToArray();

It suddenly also takes north of 2000ms to execute.

How can this be? I am completely running out of explanations or possible reasons. Anyone who could point me to what I am missing here? The slow down cannot be caused by locking/blocking because I do not alter the way the BlockingCollection is accessed between my comparisons.

Thanks for any input.

Answer 1

As with others LINQ methiods out there (and im guessing you are using LINQ’s take), this method works with a deffered execution:

This method is implemented by using deferred execution. The immediate return value is an object that stores all the information that is required to perform the action. The query represented by this method is not executed until the object is enumerated either by calling its GetEnumerator method directly or by using foreach in Visual C# or For Each in Visual Basic

This means that if you dont add the ToList() call of the foreach loop, the sole call to Take doesnt really produce any results, and the actuall results are only brought when you are using the iterator (of foreach/tolist), hence the performance difference.

Your comparison to just iterating over list might not provide accurate results; Its not the iteration over the List(foo) that takes time, its probably the selection of the elements from the blocking collection you are using that slows everything down.

MSDN claims that using regular foreach on the BlockingCollection (which is probably what happens when u r using the LINQ supplied Take, which works on IEunmerable in this case) uses a snashop of the underlying collection, and this can surely slow done the processing on huge collections.