Question

what i want to do is de-dupe a text file (against itself) based on the split. Once the de-dupe has been complete write out to a new file and keep the first result. So a basic example is. I guess the question is how do you de dupe a text file in C# based on a string split.

File 1:
Apple|Turnip3234
Apple|Tunip22
Fox|dsa34
Turtle|3423
Hamster|d34
Fox|sdw2

Result:
Apple|Turnip3234
Fox|dsa34
Turtle|3423
Hamster|d34

Answer 1

string inputFile; // = ...
string outputFile; // = ...
HashSet<string> keys = new HashSet<string>();

using (StreamReader reader = new StreamReader(inputFile))
using (StreamWriter writer = new StreamWriter(outputFile))
{
    string line = reader.ReadLine();
    while (line != null)
    {
        string candidate = line.Split('|')[0];
        if (keys.Add(candidate))
            writer.WriteLine(line);

        line = reader.ReadLine();
    } 
}