Question

The size of a file on a remote server is usually reported by the content-length header, however it is unreliable. Sometimes it is missing and occasionally it is misreported. I don’t want to exclusively rely on this header to determine whether my code should download a URL.

Is there a way to monitor the size of the file as it is downloaded and terminate the download if it exceeds a maximum size?

Answer 1

What you can do is read the max file size you want and then check whether there are more bytes to be read by doing one more read. Like this:

resp = urllib2.urlopen('http://www.google.com')
file_read = resp.read(max_wanted_size)
if resp.read(1) != '':
    #file is bigger than expected code