Home/ Questions/Q 8543469
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-11T12:18:33+00:00

I’m working on a simple proxy in python that takes a HTTP GET request

  • 0

I’m working on a simple proxy in python that takes a HTTP GET request from a browser, queries the correct website and returns the data (html, css, photos) to the client. I have it working, but it take an exorbitant amount of time to read the data back from the external web server and send it back to the client. Below is (what I think is) the relevant code:

    tempSocket.send(requestToWebpage)

    tempList = []

    while 1:
           print "waiting for data from website..."
           data =  tempSocket.recv(bufferSize)
           if not data:
                break
           else:
                tempList.append(data)

    tempResponse = ''.join(tempList)
    print "closing temp socket..."
    tempSocket.close()

    splitResponse = tempResponse.partition("\r\n")

    response = splitResponse[0] + "\r\n" + "Proxy-connection: close\r\n" + splitResponse[2]

    print "sending results back..."
    newConnection.send(response)
    newConnection.close()

The proxy is running on my own machine (as is the client browser), which is Windows 7 64-bit. I have a decent wireless connection to the internet. Currently it takes upwards of several minutes to receive the results of each GET request and transmit it to the client. By watching the print statements, I’ve noticed that most of the time seems to be spend in the while loop (especially the last loop through it), but the other print messages also take way longer to appear than it seems like they should.

Any ideas on what is going on and suggestions to improve the speed?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Marcus’s comment is probably right. The remote server is not closing its connection.

    You might be asking for this behaviour, perhaps without even realising it. What is in the request to the server, i.e. what is being sent in requestToWebpage? Are you setting a Connection: Keep-Alive header?

    Keep-Alive is the default if you are using HTTP 1.1 in the request.

    If it is not because of Keep-Alive, you may need to get the Content-Length from the reply and then you’ll know how many bytes to read.

    • 0