Question

How can I read an arbitrary file and process it “piece by piece” (meaning byte by byte or some other chunk size that would give the best read performance) without loading the entire file into memory? An example of processing would be to generate an MD5 hash of the file although the answer could apply to any operation.

I’d like to have or write this but if I can get existing code that would be great too.

(c#)

Answer 1

Here’s an example of how to read a file in chunks of 1KB without loading the entire contents into memory:

const int chunkSize = 1024; // read the file by chunks of 1KB
using (var file = File.OpenRead("foo.dat"))
{
    int bytesRead;
    var buffer = new byte[chunkSize];
    while ((bytesRead = file.Read(buffer, 0, buffer.Length)) > 0)
    {
        // TODO: Process bytesRead number of bytes from the buffer
        // not the entire buffer as the size of the buffer is 1KB
        // whereas the actual number of bytes that are read are 
        // stored in the bytesRead integer.
    }
}