Question

im trying to read in a log file in c# thats huge – approx 300mbs of raw text data. ive been testing my program on smaller files approx 1mb which stores all log messages into a string[] array and searching with contains.

however that is too slow and takes up too much memory, i will never be able to process the 300mb log file. i need a way to grep the file, which quickly filters through it finding useful data and printing the line of log information corresponding to the search.

the big question is scale, i think 300mb will be my max, but need my program to handle it. what functions, data structions, searching can i use that will scale well with speed and efficiency to read a log file that big

Answer 1

File.ReadLines is probably your best bet as it gives you an IEnumerable of lines of the text file and reads them lazily as you iterate over the IEnumerable. You can then use whatever method for searching the line you’d like to use (Regex, Contains, etc) and do something with it. My example below spawns a thread to search the line and output it to the console, but you can do just about anything. Of course, TEST, TEST, TEST on large files to see your performance mileage. I imagine if each individual thread spawned below takes too long, you can run into a thread limit.

IEnumerable<string> lines = File.ReadLines("myLargeFile.txt");
foreach (string line in lines) {
    string lineInt = line;
    (new Thread(() => {
        if (lineInt.Contains(keyword)) {
            Console.WriteLine(lineInt);
        }
    })).Start();
}

EDIT: Through my own testing, this is obviously faster:

foreach (string lineInt in File.ReadLines("myLargeFile.txt").Where(lineInt => lineInt.Contains(keyword))) {
    Console.WriteLine(lineInt);
}