Question

I’m getting a 406 error with Mechanize when trying to open a URL:

for url in urls:
    if "http://" not in url: 
        url = "http://" + url
    print url
    try:
        page = mech.open("%s" % url)
    except urllib2.HTTPError, e:
        print "there was an error opening the URL, logging it"
        print e.code
        logfile = open ("log/urlopenlog.txt", "a")
        logfile.write(url + "," + "couldn't open this page" + "\n")
        continue
    else:
        print "opening this URL..."
        page = mech.open(url)

Any idea what would cause a 406 error to occur? If I go to the URL in question I can open it in the browser.

Answer 1

Try adding headers to your request based on what your browser sends; start with adding an Accept header (406 normally means the server didn’t like what you want to accept).

See “Adding headers” in the documentation:

req = mechanize.Request(url)
req.add_header('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8')
page = mechanize.urlopen(req)

The Accept header value there is based on the header sent by Chrome.