Question

I have a LINQ expression that groups customers from an Azure Table Storage by partition.

Because Azure only supports batch operations with max 100 entities at a time (and entities in a batch much have the same PartitionKey), I need each group to contain a maximum 100 entities.

//How to complete this LINQ expression
var groups = customers.GroupBy(c => c.PartitionKey)....;

//Do some Azure table storage magic in parallel
Parallel.ForEach(groups , customersInGroup => {...});

How do I complete my LINQ expression, so each group contains max 100 customers? That is… if the customers collection eg. has 142 customers with the same PartitionKey, i want to create two groups… one groups with 100 customers and one with 42 customers.

Answer 1

There’s nothing within "normal" LINQ to do this directly, but MoreLINQ has a Batch method which you may find useful:

public static IEnumerable<TResult> Batch<TSource, TResult>
    (this IEnumerable<TSource> source, int size,
     Func<IEnumerable<TSource>, TResult> resultSelector)

public static IEnumerable<IEnumerable<TSource>> Batch<TSource>
    (this IEnumerable<TSource> source, int size)

Note that in your case you’d probably want something like:

var groups = customers.GroupBy(c => c.PartitionKey).Batch(100, p => p.ToList());

so that the returned results are materialized immediately.

Of course, this is assuming you’re using LINQ to Objects – if you’re trying to partition via another LINQ provider, I’m not sure how you’d go about it.