Question

I have multiple URLs that returns zip files. Most of the files, I’m able to download using urllib2 library as follows:

request = urllib2.urlopen(url)
zip_file = request.read()

The problem I’m having is that one of the files is 35Mb in size (zipped) and I’m never able to finish downloading it using this library. I’m able to download it using wget and the browser normally.

I have tried downloading the file in chuncks like this:

request = urllib2.urlopen(url)
buffers = []
while True:
    buffer = request.read(8192)
    if buffer:
        buffers.append(buffer)
    else:
        break
final_file = ''.join(buffers)

But this also does not finish the download. No error is raised, so it’s hard to debug what is happening. Unfortunately, I can’t post an example of the url / file here.

Any suggestions / advices?

Answer 1

This is copy / paste from my application which downloads it’s own update installer. It reads the file in blocks and immediately saves the blocks in output file on the disk.

def DownloadThreadFunc(self):
    try:
        url = self.lines[1]
        data = None
        req = urllib2.Request(url, data, {})
        handle = urllib2.urlopen(req)

        self.size = int(handle.info()["Content-Length"])
        self.actualSize = 0
        name = path.join(DIR_UPDATES, url.split("/")[-1])
        blocksize = 64*1024

        fo = open(name, "wb")
        while not self.terminate:
            block = handle.read(blocksize)
            self.actualSize += len(block)
            if len(block) == 0:
                break
            fo.write(block)
        fo.close()
    except (urllib2.URLError, socket.timeout), e:
        try:
            fo.close()
        except:
            pass
        error("Download failed.", unicode(e))  

I use self.size and self.actualSize to show the download progress in GUI thread and self.terminate to cancel the download from the GUI button if needed.