Question

Running LINQ to Objects GroupBy() method for many items (gigabytes) can be memory consuming. If the IEnumerable<T> is already ordered by the key, we could write an GroupBy that didn’t consume as much memory.

Where can I find a library that has such method?

Answer 1

There’s nothing in the framework to do this. If you don’t need an actual IGrouping<,> you could use this:

static IEnumerable<IList<TElement>> GroupByChanges<TElement, TKey>
    (this IEnumerable<TElement> source,
     Func<TElement, TKey> projection)
{
    // TODO: Argument validation, splitting this into two methods
    // to achieve eager validation.
    // TODO: Allow a custom comparer to be used, possibly even
    // an IComparer<T> instead of an IEqualityComparer<T>
    IEqualityComparer<TKey> comparer = EqualityComparer<TKey>.Default;

    using (IEnumerator<TElement> iterator = source.GetEnumerator())
    {
        if (!iterator.MoveNext())
        {
            yield break;
        }
        TKey currentKey = projection(iterator.Current);
        IList<TElement> currentList = new List<TElement> { iterator.Current };
        while (iterator.MoveNext())
        {
            TKey key = projection(iterator.Current);
            if (!comparer.Equals(currentKey, key))
            {
                yield return currentList;
                currentList = new List<TElement>();
            }
            currentList.Add(iterator.Current);
        }
        yield return currentList;
    }
}

If you need a full IGrouping<,> implementation it’ll be slightly harder – but you could always grab my Edulinq implementation.

The implementation of GroupByChanges would change very little – just change the currentList assignments to pass in the key to the Grouping constructor:

Grouping<TKey, TElement> currentGroup = new Grouping<TKey, TElement>(currentKey)
    { iterator.Current };