Question

I have a .txt file which has about 500k entries, each separated by new line. The file size is about 13MB and the format of each line is the following:

SomeText<tab>Value<tab>AnotherValue<tab>

My problem is to find a certain “string” with the input from the program, from the first column in the file, and get the corresponding Value and AnotherValue from the two columns.

The first column is not sorted, but the second and third column values in the file are actually sorted. But, this sorting is of no good use to me.

The file is static and does not change. I was thinking to use the Regex.IsMatch() here but I am not sure if that’s the best approach here to go line by line.

If the lookup time would increase drastically, I could probably go for rearranging the first column (and hence un-sorting the second & third column). Any suggestions on how to implement this approach or the above approach if required?

After locating the string, how should I fetch those two column values?

EDIT
I realized that there will be quite a bit of searches in the file for atleast oe request by the user. If I have an array of values to be found, how can I return some kind of dictionary having a corresponding values of found matches?

Answer 1

How many times do you need to do this search?
Is the cost of some pre-processing on startup worth it if you save time on each search?
Is loading all the data into memory at startup feasible?
Parse the file into objects and stick the results into a hashtable?

I don’t think Regex will help you more than any of the standard string options. You are looking for a fixed string value, not a pattern, but I stand to be corrected on that.

Update
Presuming that the “SomeText” is unique, you can use a dictionary like this

Data represents the values coming in from the file.
MyData is a class to hold them in memory.

public IEnumerable<string> Data = new List<string>() {
  "Text1\tValue1\tAnotherValue1\t",
  "Text2\tValue2\tAnotherValue2\t",
  "Text3\tValue3\tAnotherValue3\t",
  "Text4\tValue4\tAnotherValue4\t",
  "Text5\tValue5\tAnotherValue5\t",
  "Text6\tValue6\tAnotherValue6\t",
  "Text7\tValue7\tAnotherValue7\t",
  "Text8\tValue8\tAnotherValue8\t"
};

public class MyData {
   public String SomeText { get; set; }
   public String Value { get; set; }
   public String AnotherValue { get; set; }
}


[TestMethod]
public void ParseAndFind() {

        var dictionary = Data.Select(line =>
        {
            var pieces = line.Split(new char[] { '\t' }, StringSplitOptions.RemoveEmptyEntries);
            return new MyData {
                SomeText = pieces[0],
                Value = pieces[1],
                AnotherValue = pieces[2],
            };
        }).ToDictionary<MyData, string>(dat =>dat.SomeText);

        Assert.AreEqual("AnotherValue3", dictionary["Text3"].AnotherValue);
        Assert.AreEqual("Value7", dictionary["Text7"].Value);

}

hth,

Alan