Question

The title doesn’t make it that obvious what I’m asking, but I have created an algorithm that compresses a bunch of files into a single file, and then decompresses them again. To avoid OutOfMemory Exceptions, I use two fileStreams to, first read segments of data from the original files, and then the other fileStream to write these segments into the final file.

I have included my code excerpt below. In this case, rStream and wStream are already declared accordingly, and the bufferSize is currently at 16 mB. fInfo is the file info for the file we are reading from.

Obviously the higher the bufferSize, the faster the operation is completed. I want to know what the maximum possible bufferSize I should use to maximize the efficiency of the operation?

                int bytesRead = 0;
                long toRead = fInfo.Length - curFileSize;
                if (toRead > bufferSize) { toRead = bufferSize; }
                byte[] fileSegment = new byte[toRead];

                while (bytesRead < toRead)
                {
                    bytesRead += rStream.Read(fileSegment, bytesRead, (int)toRead - bytesRead);
                }

                wStream.Seek(finalFileSize, SeekOrigin.Begin);
                wStream.Write(fileSegment, 0, (int)toRead);

Answer 1

A buffer of 16 MB definitely sounds like overkill. Usually a few kilobytes is used for a buffer like that. At 16 MB you would have very little gain in making the buffer larger, or none at all.

Consider that if you are using a large buffer, it won’t fit in the processor cache and would be slower to access. If you make it really large some part of it may even be swapped out to disk, so at that point making the buffer larger would only make it slower.