Question

I am playing around with sockets in Python (to learn them), and I am learning how HTTP protocol works while I am at it. I was doing fine until I learned there is something called Transfer-Encoding: chunked. I did a google search on it, and found this wikipedia article. It says size of each chunk is specified before the chunk itself. But it doesn’t say how many bytes should I read from socket in order to get it correctly. Furthermore, it doesn’t say if should I care about endianness. Can anyone please provide details on this?

Answer 1

Don’t overthink it, as most things in HTTP it’s just plain text. You just need to read from the socket until you encounter a CRLF sequence, then extract the chunk size (which may be terminated by a semicolon) and interpret it as hex number.

Once you have the line this should work to extract the chunk size:

chunk_size = int(line.strip().split(';')[0], 16)