Question

I wrote a raw TCP client for HTTP/HTTPS requests, however I’m having problems with chunked encoding responses. HTTP/1.1 is requirement therefore I should support it.

Raw TCP is a business requirement that I need to keep, therefore I can’t switch to .NET HTTPWebRequest/HTTPWebResponse However if there is way to convert a RAW HTTP Request/Response into HTTPWebRequest/HTTPWebResponse that’d work.

Answer 1

The best place to start is the http 1.1 specification, which lays out how chunking works. Specifically section 3.6.1.

3.6.1 Chunked Transfer Coding

The chunked encoding modifies the body of a message in order to
transfer it as a series of chunks, each with its own size indicator,
followed by an OPTIONAL trailer containing entity-header fields. This allows dynamically produced content to be transferred along with the
information necessary for the recipient to verify that it has
received the full message.

   Chunked-Body   = *chunk                     last-chunk                     trailer                     CRLF     chunk          = chunk-size [ chunk-extension ] CRLF                     chunk-data CRLF    chunk-size     = 1*HEX    last-chunk     = 1*('0') [ chunk-extension ] CRLF     chunk-extension= *( ';' chunk-ext-name [ '=' chunk-ext-val ] )    chunk-ext-name = token    chunk-ext-val  = token | quoted-string    chunk-data     = chunk-size(OCTET)    trailer        = *(entity-header CRLF) 

The chunk-size field is a string of hex digits indicating the size of
the chunk. The chunked encoding is ended by any chunk whose size is
zero, followed by the trailer, which is terminated by an empty line.

The trailer allows the sender to include additional HTTP header
fields at the end of the message. The Trailer header field can be used to indicate which header fields are included in a trailer (see section 14.40).

Assuming that you have already read the header from the response and are pointing to the next byte in the stream your pseudo code would look something like this:

done = false; uint8 bytes[]; while (!done) {   chunksizeString = readuntilCRLF(); // read in the chunksize as a string   chunksizeString.strip(); // strip off the CRLF   chunksize = chunksizeString.convertHexString2Int(); // convert the hex string to an integer.   bytes.append(readXBytes(chunksize)); // read in the x bytes and append them to your buffer.   readCRLF(); // read the trailing CRLF and throw it away.   if (chunksize == 0)      done = true; //  } // now read the trailer if any // trailer is optional, so it may be just the empty string trailer = readuntilCRLF() trailer = trailer.strip() if (trailer != '')    readCRLF(); // read out the last CRLF and we are done. 

This is ignoring the chunk-extension portion, but since it is delimited with a ‘;’ it should be easy to split it out. This should be enough to get you started. Keep in mind that the chunksize string does not have a leading ‘0x’.