Home/ Questions/Q 8439867
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-10T08:07:35+00:00

I’m attempting to send an HTTP Request to a website and read the data

  • 0

I’m attempting to send an HTTP Request to a website and read the data it returns. The first website I tried worked successfully. It returned about 4 packets of data and then returned a 0 packet which the script caught and terminated.

However, attempting to load http://www.google.com/ does not work this way. Instead, it returns about 10 packets of the same length, a final smaller packet, and then proceeds to time out. Is it normal for this to happen? Does it all just depend on what server the host is using?

If anyone could recommend an alternative way to reading with socket.recv() that would take into account that a final null packet is not always sent, it would be greatly appreciated. Thanks.

    try:
        data = s.recv(4096)

        while True:
            more = s.recv(4096)
            print len(more)
            if not more:
                break
            else:
                data += more
    except socket.timeout:
        errMsg = "Connection timed-out while connecting to %s. Request headers were as     follows: %s", (parsedUrl.netloc, rHeader.headerContent)
        self.logger.exception(errMsg)
        raise Exception
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    For HTTP, use requests rather than writing your own.

    > ipython

In [1]: import requests

In [2]: r = requests.get('http://www.google.com')

In [3]: r.status_code
Out[3]: 200

In [4]: r.text[:80]
Out[4]: u'<!doctype html><html itemscope="itemscope" itemtype="http://schema.org/WebPage">'

In [5]: len(r.text)
Out[5]: 10969
    • 0