Question

I’ve been playing around with what I thought was a simple idea. I want to be able to read in a file from somewhere (website, filesystem, ftp), perform some operations on it (compress, encrypt, etc.) and then save it somewhere (somewhere may be a filesystem, ftp, or whatever). It’s a basic pipeline design. What I would like to do is to read in the file and put it onto a MemoryStream, then perform the operations on the data in the MemoryStream, and then save that data in the MemoryStream somewhere. I was thinking I could use the same Stream to do this but run into a couple of problems:

1. Everytime I use a StreamWriter or StreamReader I need to close it and that closes the stream so that I cannot use it anymore. That seems like there must be some way to get around that.
2. Some of these files may be big and so I may run out of memory if I try to read the whole thing in at once.

I was hoping to be able to spin up each of the steps as separate threads and have the compression step begin as soon as there is data on the stream, and then as soon as the compression has some compressed data available on the stream I could start saving it (for example). Is anything like this easily possible with the C# Streams? ANyone have thoughts as to how to accomplish this best?

Thanks,

Mike

Answer 1

Using a helper method to drive the streaming:

static public void StreamCopy(Stream source, Stream target)
{
    byte[] buffer = new byte[8 * 1024];

    int size;
    do
    {
      size = source.Read(buffer, 0, 8 * 1024);
      target.Write(buffer, 0, size);
    } while (size > 0);
}

You can easily combine whatever you need:

using (FileStream iFile = new FileStream(...))
using (FileStream oFile = new FileStream(...))
using (DeflateStream oZip = new DeflateStream(outFile, CompressionMode.Compress))
    StreamCopy(iFile, oZip);

Depending on what you are actually trying to do, you’d chain the streams differently. This also uses relatively little memory, because only the data being operated upon is in memory.