Question

after reading this very interesting thread on duplicate removal, i ended with this =>

    public static IEnumerable<T> deDuplicateCollection<T>(IEnumerable<T> input)
    {
        var hs = new HashSet<T>();
        foreach (T t in input)
            if (hs.Add(t))
                yield return t;
    }        

by the way, as i’m brand new to C# and coming from Python, i’m a bit lost between casting and this kind of thing… i was able to compile and build with :

            foreach (KeyValuePair<long, List<string>> kvp in d)
            {
                d[kvp.Key] = (List<string>) deDuplicateCollection(kvp.Value);
            }

but i must have missed something here… as i get a “System.InvalidCastException” @ runtime, maybe could you point interesting things about casting and where i’m wrong? Thank you in advance.

Answer 1

First, about the usage of the method.

Drop the cast, invoke ToList() on the result of the method. The result of the method is IEnumerable<string>, this is not a List<string>. The fact the source is originally a List<string> is irrelevant, you don’t return the list, you yield return a sequence.

d[kvp.Key] = deDuplicateCollection(kvp.Value).ToList();

Second, your deDuplicateCollection method is redundant, Distinct() already exists in the library and performs the same function.

d[kvp.Key] = kvp.Value.Distinct().ToList();

Just be sure you have a using System.Linq; in the directives so you can use these Distinct() and ToList() extension methods.

Finally, you’ll notice making this change alone, you run into a new exception when trying to change the dictionary in the loop. You cannot update the collection in a foreach. The simplest way to do what you want is to omit the explicit loop entirely. Consider

d = d.ToDictionary(kvp => kvp.Key, kvp => kvp.Value.Distinct().ToList());

This uses another Linq extension method, ToDictionary(). Note: this creates a new dictionary in memory and updates d to reference it. If you need to preserve the original dictionary as referenced by d, then you would need to approach this another way. A simple option here is to build a dictionary to shadow d, and then update d with it.

var shadow = new Dictionary<string, string>();
foreach (var kvp in d)
{ 
    shadow[kvp.Key] = kvp.Value.Distinct().ToList();
}

foreach (var kvp in shadow)
{
    d[kvp.Key] = kvp.Value;
}

These two loops are safe, but you see you need to loop twice to avoid the problem of updating the original collection while enumerating over it while also preserving the original collection in memory.