Question

I have written a c# win forms application that allows the user to open a log (text) file and view the log lines in a data grid. The application formats that log data so that the user can filter, search etc.

The problem I have is that when the user opens a log file > 300mb the application throws an out of memory exception.

The app first loads all of the log lines into a string array, then it loops through the log lines, adding log entry objects to a list.

var allLogLines = File.ReadAllLines(logPath).ToList();
var nonNullLogLines = allLogLines.Where(l => !string.IsNullOrEmpty(l));

this.ParseLogEntries(nonNullLogLines.ToArray());

This initial step (loading the log data into a string array) uses up about 1gb of memory in task manager.

internal override void ParseLogEntries(string[] logLines)
{
    this.LogEntries = new List<LogEntry>();
    this.LogLinesCount = logLines.Count();

    for (int i = 0; i < this.LogLinesCount; i++)
    {
        int entryStart = this.FindMessageCompartment(logLines, i);
        int entryEnd = this.FindMessageCompartment(logLines, entryStart + 1);
        int entryLength = (entryEnd - entryStart) + 1;

        if (entryStart + entryLength > this.LogLinesCount)
        {
            entryLength = this.LogLinesCount - entryStart;
        }

        var logSection = new string[entryLength];

        Array.Copy(logLines, entryStart, logSection, 0, entryLength);
        Array.Clear(logLines, i, entryLength - 1);

        this.AddLogEntry(logSection);

        i = (entryEnd - 1);
    }
}

The AddLogEntry method addes a log entry to the list (LogEntries). The for loop manages to parse about 50% of the log file, then the out of memory exception occurs. At this point task manager reports that the application is using about 1.3gb of memory.

As you can see above I have added Array.Clear to null out the portion of the log data that has been successfully parsed, as a result I would expect that as objects are being added to the collection, the amount of memory (1gb to begin with) used by the large log data array would steadily reduce, but it does not. in fact this line makes no difference to the memory usage, even if I add a GC collect periodically.

Having read about LOH, I am assuming this is because the heap is not being compressed as portions of the large array are being nulled, so it always uses the same 1gb of memory despite its contents.

Is there any way I can reduce the amount of memory being held while the data is being parsed, or a possible rework that may make better use of memory? It seems strange to me that a 300mb text file, when put into a string array, consumes 1gb of memory?

Thanks.

Answer 1

Instead of your method ParseLogEntries(string[] logLines) that parses all the log lines in one go, you could instead have a ParseLogEntry(string logLine) method that parses a single line.

If you combine this with iterating over the lines in your log file one at a time (for instance by creating yourself an enumerator), this would avoid creating the big array string[] logLines in the first place.

One way could be like this:

static IEnumerable<string> ReadLines(string filename)
{
    using (TextReader reader = File.OpenText(filename))
    {
        string line;
        while ((line = reader.ReadLine()) != null)
        {
            yield return line;
        }
    }
}

// And use the function somewhere to parse the log

var logEntries = new List<LogEntry>()
foreach (string line in ReadLines("log.txt"))
{
    logEntries.Add(ParseLogEntry(line));
}

If you’re using .NET 4.0 or greater, you could of course just use the File.ReadLines Method as pointed out by sll in another answer, instead of creating your own method.