Question

I’m working with Python’s Mechanize module. I’ve come across 3 different sites that cannot be opened by mechanize directly:

1. en.wikipedia.org/wiki/Dog (new user, can’t post more than 2 links T-T )
2. https://www.google.com/search?num=100&hl=en&site=&q=dog&oq=dog&aq=f&aqi=g10&aql=1&gs_sm=e&gs_upl=618l914l0l1027l3l2l0l0l0l0l173l173l0.1l1l0

3. http://www.cpsc.gov/cpscpub/prerel/prhtml03/03059.html

   import mechanize
   br = mechanize.Browser()
   br.set_handle_robots(False)
   

Adding the following code allows mechanize to open and parse the wikipedia article and the google search results:

    br.addheaders = [('User-agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1')] 

But, my workarounds are no match for the CPSC.gov website – when I try to open it with the mechanize Browser, my python freezes – to the point where I can’t even Keyboard Interrupt it.

What’s going on here?

Answer 1

In the case of the cpsc.gov site, it looks like there’s a refresh header that isn’t being correctly processed by mechanize HTTPRefreshProcessor. However, you can workaround the problem as follows:

import mechanize

url = 'http://www.cpsc.gov/cpscpub/prerel/prhtml03/03059.html'
br = mechanize.Browser()
br.set_handle_refresh(False)
br.open(url)