Question

I’m using the System.Net.HttpWebRequest class to implement a simple HTTP downloader that can be paused, canceled and even resumed after it was canceled (with the HTTP Range request header).

It’s clear that HttpWebRequest.GetResponse() is when the HTTP request is actually sent to the server, and the method returns when a HTTP response is received (or a timeout occurs). However, the response body is represented with a Stream, which leaves me wonder whether the response body is actually transmitted with the response header (i.e. it’s already downloaded when GetResponse() returns), or is it only downloaded on-demand, when I try to read from the response stream? Or maybe when I call the HttpWebResponse.GetResponseStream() method?

Unfortunately the msdn documentation doesn’t tell, and I don’t know enough about the HTTP protocol to be able to tell.

How do chunked transfers and the like behave in this case (that is, how should I handle them in my C# application)? When is actually the response data downloaded from the server?

Answer 1

This all depends on TCP, the underlying protocol of HTTP. The way TCP works is that data is sent in segments. Whenever a client sends a segment to the server, among the data sent is information about how much additional data is it ready to receive. This usually corresponds to some kind of buffer on the client’s part. When the client receives some data, it also sends a segment to the server, acknowledging the received data.

So, assuming the client is very slow in processing the received data, the sequence of events could go like this:

1. Connection is established, the clients says how much data is it ready to receive.
2. Server sends one or more segments to the client, the total data in them at most the amount client said it is ready to receive
3. Client says to the server: I received the data you sent me, but don’t send me anymore for now.
4. Client processes some of the data.
5. Client says to the server: You can send me x more bytes of data

What does this mean with regards to GetResponse()? When you call GetResponse(), the client sends the request, reads the HTTP header of the response (which usually fits into one segment, but it may be more) and returns. At this point, if you don’t start reading the response stream (that you get by calling GetResponseStream()), some data from the server is received, but only to fill the buffer. When that is full, no more data is transmitted until you start reading the response stream.