Question

I need to read line by line from text file (log files from server) and they are big (about 150-200MB). I am using StreamReader and its great for “little” files like 12MB but not for so big. After sometime it is loaded and it shows in my DataGridView but its huge in memory. I am using bindingSource.Filter on this DataGridView (like textbox and when user write letter it is filtering one column a comparing strings, not showing rows without letters in textbox and so) and with big files its useless too. So I want to ask you what is best solution for me.

I was looking and find some solutions but I need help with decided whats best for me and with implementing (or if there is something else):

1. Load data in background and showing them in realtime. I am not really sure how to do that and I don´t know what to do with filtering in this solution.
2. Maybe upgrade somehow streamreader? Or write own method for reading lines from file with binary readers?
3. I found something about Memory-Mapped in c# 4.0 but i can´t use 4.0. Could this help feature help?

Thanks for help

---

Okay, so I am implementing Paging and I read 5k lines of text file than after clicking button next lines and so. I am using BaseStream.Position for saving a starting reading but I would like to use some other function which save number of lines and mainly I want use method for starting reading from exact line but I can´t find nothing for StreamReader. Is there something like that?

Answer 1

Load data in background and showing them in realtime. I am not really sure how to do that and I don’t know what to do with filtering in this solution.

This is no help. It will still consume much memory in the background thread.

Maybe upgrade somehow streamreader? Or write own method for reading lines from file with binary readers?

Still no help, once you read the whole file into memory it will, well, consume memory.

I think you get the point. Don’t load the whole file into memory. Load only chunks of it. Use paging. You cannot show 200MB worth of data on a single screen anyways, so only load the portion you need to show on the screen. So basically you need to implement the following function:

public IEnumerable<string> ReadFile(int page, int linesPerPage, out totalLines)
{
    ...
}

The Skip and Take extension methods could be helpful here.