Question

I don’t know if I’m doing something wrong, but I’m 100% sure it’s the python script brings down my Internet connection.

I wrote a python script to scrape thousands of files header info, mainly for Content-Length to get the exact size of each file, using HEAD request.

Sample code:

class HeadRequest(urllib2.Request):
    def get_method(self):
        return "HEAD"

response = urllib2.urlopen(HeadRequest("http://www.google.com"))
print response.info()

The thing is after several hours running, the script starts to throw out urlopen error timed out, and my Internet connection is down from then on. And my Internet connection will always be back on immediately after I close that script. At the beginning I thought it might be the connection not stable, but after several times running, it turned out to be the scripts fault.

I don’t know why, this should be considered as a bug, right? Or my ISP banned me for doing such things? (I already set the program to wait 10s each request)

BTW, I’m using VPN network, does it have something to do with this?

Answer 1

I’d guess that either your ISP or VPN provider is limiting you because of high-volume suspicious traffic, or your router or VPN tunnel is getting clogged up with half-open connections. Consumer internet is REALLY not intended for spider-type activities.