Question

I’m having a hard time deduping a list based on a specific delimiter.

For example I have 4 strings like below:

apple|pear|fruit|basket
orange|mango|fruit|turtle
purple|red|black|green
hero|thor|ironman|hulk

In this example I should want my list to only have unique values in column 3, so it would result in an List that looks like this,

apple|pear|fruit|basket
purple|red|black|green
hero|thor|ironman|hulk

In the above example I would have gotten rid of line 2 because line 1 had the same result in column 3. Any help would be awesome, deduping is tough in C#.

how i’m testing this:

    static void Main(string[] args)
    {
        BeginListSet = new List<string>();
        startHashSet();
    }


    public static List<string> BeginListSet { get; set; }

    public static void startHashSet()
    {
        string[] BeginFileLine = File.ReadAllLines(@"C:\testit.txt");
        foreach (string begLine in BeginFileLine)
        {

            BeginListSet.Add(begLine);
        }

    }

    public static IEnumerable<string> Dedupe(IEnumerable<string> list, char seperator, int keyIndex)
    {
        var hashset = new HashSet<string>();
        foreach (string item in list)
        {
            var array = item.Split(seperator);
            if (hashset.Add(array[keyIndex]))
                yield return item;
        }
    }

Answer 1

Try this:

var list = new string[]
                    {
                        "apple|pear|fruit|basket",
                        "orange|mango|fruit|turtle",
                        "purple|red|black|green",
                        "hero|thor|ironman|hulk "
                    };

var dedup  = new List<string>();
var filtered = new List<string>();
foreach (var s in list)
{
    var filter = s.Split('|')[2];
    if (dedup.Contains(filter)) continue;
    filtered.Add(s);
    dedup.Add(filter);
}


// Console.WriteLine(filtered);