Question

I am writing an ASP.NET Web API application that requires me to accept a file upload and forward that file on to another HTTP endpoint.

I am concerned that if many users try to upload a 100MB file each (which is a valid use case) then my application will have a large memory footprint and depending on the volume of large requests this footprint could grow to large and my application would keel over and die.

Ideally I’d like begin streaming the file to the other HTTP end point as soon as the web server starts to receive the file to significantly reduce the load on the server.

I’m sure this process has a name but I don’t know it – which is making searching for it rather tough.

I’ve done quite a bit of work with Response Streaming in the Web API but I’ve never had to consider request streaming before.

Best I can tell I need to work out how to:

• Begin processing the stream before it has finished uploading.
• Use the HttpClient to stream the same request to stream the same data to the other HTTP end point.

Can anyone offer me some pointers?

Answer 1

That’s an interesting question. I’ll try to do my best to give some general pointers.

Few things to consider:

1) Web API by default buffers requests so your fear that the memory footprint might be considerable is definitely justified. You can force Web API to work with requests in a streamed mode:

    public class NoBufferPolicySelector : WebHostBufferPolicySelector
    {
       public override bool UseBufferedInputStream(object hostContext)
       {
          var context = hostContext as HttpContextBase;

          if (context != null)
          {
             if (string.Equals(context.Request.RequestContext.RouteData.Values["controller"].ToString(), "uploading", StringComparison.InvariantCultureIgnoreCase))
                return false;
          }

          return true;
       }

       public override bool UseBufferedOutputStream(HttpResponseMessage response)
       {
          return base.UseBufferedOutputStream(response);
       }
    }

And then replace the service:

GlobalConfiguration.Configuration.Services.Replace(typeof(IHostBufferPolicySelector), new NoBufferPolicySelector());

Please note that due to differences between WebHost and SelfHost at this point, such change is only possible in WebHost. If your endpoint is selfHosted, you would have to set the streaming mode at the GlobalConfig level:

//requests only
selfHostConf.TransferMode = TransferMode.StreamedRequest;
//responses only
selfHostConf.TransferMode = TransferMode.StreamedResponse;
//both
selfHostConf.TransferMode = TransferMode.Streamed;

I have blogged about dealing with large files in Web API in more details before – http://www.strathweb.com/2012/09/dealing-with-large-files-in-asp-net-web-api/ so hopefully you’ll find that useful.

2) Secondly, if you use HttpClient, in .NET 4 it buffers the requests body by default, so you should really use .NEt 4.5.

If you have to use .NET 4 you have to work with HttWebRequest directly:
– http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.allowreadstreambuffering.aspx
– http://msdn.microsoft.com/en-us/library/system.net.httpwebrequest.allowwritestreambuffering.aspx

3) As far as pushing the data to the client that’s definitely possible if you want to do that, with PushStreamContent.
Henrik has a short introductory post here – http://blogs.msdn.com/b/henrikn/archive/2012/04/23/using-cookies-with-asp-net-web-api.aspx (it’s based on Web API RC bits so you might need to adjust some signatures etc.)
I also blogged about pushing chunks of stream data here – http://www.strathweb.com/2013/01/asynchronously-streaming-video-with-asp-net-web-api/

EDIT: To see an example if PushStreamContent in the request, you can have a look at this sample solution – http://aspnet.codeplex.com/SourceControl/changeset/view/bb167f0b0013#Samples/Net45/CS/WebApi/UploadXDocumentSample/ReadMe.txt