Question

I am attempting to do the following:

request = urllib2.Request(url=url, headers={ 'User-Agent' : 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT' })
response = urllib2.urlopen(request)
HTML_response = response.read()
response.close()
return BeautifulSoup(HTML_response)

however, on some pages (always the same pages, but it does not look like order is an issue) I get

Traceback (most recent call last):
  File "/usr/lib/python2.7/multiprocessing/queues.py", line 268, in _feed
    send(obj)
  File "/usr/local/lib/python2.7/dist-packages/BeautifulSoup.py", line 439, in __getnewargs__
    return (NavigableString.__str__(self),)
RuntimeError: maximum recursion depth exceeded while calling a Python object

which does exist, so I don’t think doing except urllib2.HTTPError: will help

Answer 1

In [1]: import urllib2

In [2]: from BeautifulSoup import BeautifulSoup

In [3]: url = 'http://www.sparklebox.co.uk/topic/creative-arts/art-and-design/colouring-pages.html'

In [4]: request = urllib2.Request(url=url, headers={ 'User-Agent' : 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT' })

In [5]: response = urllib2.urlopen(request)    
In [6]: HTML_response = response.read()    
In [7]: b1 = BeautifulSoup(HTML_response)    
In [8]: print type(b1)
<class 'BeautifulSoup.BeautifulSoup'>

Its working fine with BeautifulSoup 3.2