Question

I am pasting whole code, since I think it is as short as it can get;

import socket

def recv_all(sock):
    parts = []
    while True:
        msg = ""
        msg = sock.recv(4096)
        print "recieved", len(msg), "bytes."
        if not msg:
            break
        parts.append(msg)
    return "".join(parts)

http_request = """GET / HTTP/1.1
Host: {}
User-Agent: Opera/9.80 (Windows NT 6.1; WOW64; U; tr) Presto/2.10.229 Version/11.64
Accept: text/html, application/xml;q=0.9, application/xhtml xml, image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1
Accept-Language: en
Accept-Charset: iso-8859-1, utf-8, utf-16, *;q=0.1
Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0
Connection: Keep-Alive, TE
TE: deflate, gzip, chunked, identity, trailers


"""
site = "www.google.com.tr"
print "creating connection"
conn = socket.create_connection((site,80))
print "sending data"
conn.sendall(http_request.format(site))
print "recieving response"
response = recv_all(conn)
print "closing connection"
conn.close()

print response

When I run this, I get this output;

creating connection
sending data
recieving response
recieved 3743 bytes.
recieved 4096 bytes.
recieved 4096 bytes.
recieved 4096 bytes.
recieved 3472 bytes.
recieved 1648 bytes.

What I expected it to do is to read 0 bytes after it reads last chunk (1648 in this case). But it keeps waiting instead.

I am on windows 7 with Python 2.7.

What am I doing wrong here? What should I do so that it works as intended?

Answer 1

You’re sending:

Connection: Keep-Alive, TE

… and the server’s doing exactly as you asked – keeping the connection alive at the end. So your read call blocks, waiting for more data – while the server is waiting for another request on the same connection.

Either turn off keep-alive, or look for a Content-Length header, and stop reading when you’ve read that much data.

(Alternatively, use an HTTP client library instead of trying to do it yourself 🙂