Question

I am using urllib’s urlretrieve() function in Python in order to try to grab some pdf’s from websites. It has (at least for me) stopped working and is downloading damaged data (15 KB instead of 164 KB).

I have tested this with several pdf’s, all with no success (ie random.pdf). I can’t seem to get it to work, and I need to be able to download pdf’s for the project I am working on.

Here is an example of the kind of code I am using to download the pdf’s (and parse the text using pdftotext.exe):

def get_html(url): # gets html of page from Internet
    import os
    import urllib2
    import urllib
    from subprocess import call
    f_name = url.split('/')[-2] # get file name (url must end with '/')
    try:
        if f_name.split('.')[-1] == 'pdf': # file type
            urllib.urlretrieve(url, os.getcwd() + '\\' + f_name)
            call([os.getcwd() + '\\pdftotext.exe', os.getcwd() + '\\' + f_name]) # use xpdf to output .txt file
            return open(os.getcwd() + '\\' + f_name.split('.')[0] + '.txt').read()
        else:
            return urllib2.urlopen(url).read()
    except:
        print 'bad link: ' + url    
        return ""

I am a novice programmer, so any input would be great! Thanks

Answer 1

I would suggest trying out requests. It is a really nice library that hides all of the implementation behind a simple api.

>>> import requests
>>> req = requests.get("http://www.mathworks.com/moler/random.pdf")
>>> len(req.content)
167633
>>> req.headers
{'content-length': '167633', 'accept-ranges': 'bytes', 'server': 'Apache/2.2.3 (Red Hat) mod_jk/1.2.31 PHP/5.3.13 Phusion_Passenger/3.0.9 mod_perl/2.0.4 Perl/v5.8.8', 'last-modified': 'Fri, 15 Feb 2008 17:11:12 GMT', 'connection': 'keep-alive', 'etag': '"30863b-28ed1-446357e3d4c00"', 'date': 'Sun, 03 Feb 2013 05:53:21 GMT', 'content-type': 'application/pdf'}

By the way, the reason you are only getting a 15kb download is because your url is wrong. It should be

http://www.mathworks.com/moler/random.pdf

But you are GETing

http://www.mathworks.com/moler/random.pdf/

>>> import requests
>>> c = requests.get("http://www.mathworks.com/moler/random.pdf/")
>>> len(c.content)
14390