Question

Is there a pattern to combine parallel with a thread safe calculation on the parallel?

Need to calculate a result in which the first step would benefit from parallel and the second is a serial process on the results of the parallel.

One option is to run the parallel and save the output to a collection and then serially process the collection and I have that working. The problem there is memory management as the collection can be very large.

Below is the serial version. Basially I want to parallel the TableQueryGetRowKeys and use that result in a thread safe manner. Tried to just Parallel the for and put a lock around the final results but rowKeys could be off. Tried aggregate but I could not figure out how to pass a collection to the aggregate let alone perform thread safe Intersect in the aggregate.

IEnumerable<string> finalResults = null;
if (partitionKey.Length == 0) return finalResults;
object lockObject = new object();
finalResults = TableQueryGetRowKeys(partitionKey[0], 0);
HashSet<string> rowKeys;
for(int i = 1; i < partitionKey.Length; i++)
{
    // IO operation to Azure Table Storage against the PartitionKey
    // so very amenable to parallel
    rowKeys = TableQueryGetRowKeys(partitionKey[i]);
    // a memory and CPU operation 
    // this should be much faster than TableQueryGetRowKeys
    // going parallel and wrapping this in a lock did not properly synch rowKeys
    finalResults = finalResults.Intersect(rowKeys); 
}
return finalResults;

Answer 1

Assuming that TableQueryGetRowKeys is thread safe:

var final = partitionKey.AsParallel()
                        // By returning AsParallel we can get parallel intersect
                        .Select(k => TableQueryGetRowKeys(k).AsParallel())
                        .Aggregate((x, y) => x.Intersect(y));

// Using fake-ish data I see about a 30% speed-up on a 4-core machine:
// static HashSet<string> TableQueryGetRowKeys(string prefix)
// {
//     // Simulate 1s of IO round-trip
//     if (useSleep) Thread.Sleep(1000);
//
//     return new HashSet<string>(
//         Enumerable.Range(0, 500)
//                   .Select(_ => random.Value.Next(0, 500).ToString()));
// }

In stepwise fashion this algorithm works like so:

1. partitionKey.AsParallel() turns the regular IEnumerable<string> into a ParallelQuery<string> which allows parallel processing of the sequence.
2. Next, ParallelEnumerable.Select is used to call TableQueryGetRowKeys in parallel.
3. The result of each call to TableQueryGetRowKeys is then wrapped in a ParallelQuery<T> using AsParallel().
4. ParallelEnumerable.Intersect is used as an aggregation function over each “parallel-enabled” enumeration returned by TableQueryGetRowKeys.

---

In effect, this could be used in serial to replace your previous code by removing the AsParallel calls, like so:

var serialEquivalent = partitionKey.Select(k => TableQueryGetRowKeys(k))
                                   .Aggregate((x,y) => x.Intersect(y));

You can “convince” yourself that this is equivalent to your method when you look at the meat and potatoes of your implementation:

IEnumerable<string> results = SomeMethod(0);
for (int ii = 1; ii < count; ++ii)
{
    results = results.Intersect(SomeMethod(ii));
}

Rewriting the above using + instead of Intersect:

int results = SomeMethod(0);
for (int ii = 1; ii < count; ++ii)
{
    results = results + SomeMethod(ii);
}

Now it becomes clear that Intersect could be used in place of other more “common” aggregation functions (e.g. mathematical operators).