Question

I have a log file that can get pretty big.

The information in my log file is in a certain format and I want to be retreiving them a seperate blocks of data.

For example,

This is the start.

Blah Blah

Blah Blah Blah Blah Blah Blah

Blah

This is the start.

Blah Blah

Blah Blah Blah Blah Blah Blah

Blah Blah Blah Blah Blah Blah

Blah Blah Blah Blah Blah Blah

Blah

I want to get information from the “this is the start” to before the start of next “this is the start”. What is the best way to do this? My code is in c#.

Answer 1

The following code will split the file into chunks delineated by the "This is the start." line and call a callback method to process each chunk:

public static void ProcessInChunks(string inputFilename,
    string delimiter, Action<IEnumerable<string>> processChunk)
{
    using (var enumerator = File.ReadLines(inputFilename).GetEnumerator())
    {
        if (!enumerator.MoveNext())
            // The file is empty.
            return;

        var firstLine = enumerator.Current;
        if (firstLine != delimiter)
            throw new InvalidOperationException(
                "Expected the first line to be a delimiter.");

        List<string> currentChunk = new List<string>();

        while (enumerator.MoveNext())
        {
            if (enumerator.Current == delimiter)
            {
                processChunk(currentChunk);
                currentChunk = new List<string>();
            }
            else
                currentChunk.Add(enumerator.Current);
        }
        processChunk(currentChunk);
    }

Usage:

ProcessInChunks(@"myfile.log", "This is the start.",
    chunk => { /* do something here */ });