Question

I am looking to improve a WCF Client/Server so that it will deal with large numbers of small files, faster than it currently does.

I have written a WCF client and a server to move files across a network.

I have it working by making a call from the client to the server (sending the name of a file that I want to download as a parameter), and then having the server returning a Stream

Simplified Example:

//CLIENT CODE>>
Stream stream = syncService.GetStream(fileName);
//<<CLIENT CODE

//SERVER CODE>>
public Stream GetStream(string fileName)
{
   string filePathOnServer = ServerService.Service1.SERVER_FILES_PATH + fileName;
   return File.OpenRead(filePathOnServer);
}
//<<SERVER CODE

I then call GetStream recursively if I need to get several files, and save the streams to Files on the Client machine.
It works acceptably when moving small numbers of large files
The issue I have is, the overhead of downloading a single file is about 1/10 of a second, regardless of size; so if I want to download a huge number of 1Kb files, I am essentially capped to a maximum of 10Kbs.

I am hoping that someone has a suggestion for an alternate implementation.
I have tried returning a List of Streams from the Server, but I gather that WCF won’t allow this.

I need to be able to do this without zipping the files.

I was considering trying to return one stream that was made up of several streams concatenated, but I’m unsure if there is a better approach.

Answer 1

I’d change your WCF method to accept a collection of file names(i.e. List<string> or string[]) , then pack them up. I know SharpZipLib works well in producing ZIP files.

You stream the ZIP file back to the client, which would in turn unpack it and process the files.

One bigger file should be orders of magnitude faster to transfer and lighter on the bandwidth (because you’d deal with less network-related overhead), not to mention the fact that you would have a single WCF invocation rather than one-per-file (a huge bottleneck).