Question

I wrote a little wrapper using beautifulsoup great html parser

recently I tried to improve the code and make all beautifulsoup methods available directly in the wrapper class (instead of through a class property ) and I thought subclassing the beautifulsoup parser would be the best way to achieve this.

Here is the the class:

class ScrapeInputError(Exception):pass
from BeautifulSoup import BeautifulSoup

class Scrape(BeautifulSoup):
    """base class to be subclassed
    basically a subclassed BeautifulSoup wrapper that providers
    basic url fetching with urllib2
    and the basic html parsing with beautifulsoup
    and some basic cleaning of head,scripts etc'"""

    def __init__(self,file):
        self._file = file
        #very basic input validation
        import re
        if not re.search(r"^http://",self._file):
            raise ScrapeInputError,"please enter a url that starts with http://"

        import urllib2
        #from BeautifulSoup import BeautifulSoup
        self._page = urllib2.urlopen(self._file) #fetching the page
        BeautifulSoup.__init__(self,self._page)
        #self._soup = BeautifulSoup(self._page) #calling the html parser

this way I can just initiate the class with

x = Scrape("http://someurl.com")

and be able to traverse the tree with x.elem or x.find

this works wonderfull with some beautifulsoup methods (see above) but fails with others – those using iterator like “for e in x:”

the error message:

 Traceback (most recent call last):
  File "<pyshell#86>", line 2, in <module>
    print e
  File "C:\Python27\lib\idlelib\rpc.py", line 595, in __call__
    value = self.sockio.remotecall(self.oid, self.name, args, kwargs)
  File "C:\Python27\lib\idlelib\rpc.py", line 210, in remotecall
    seq = self.asynccall(oid, methodname, args, kwargs)
  File "C:\Python27\lib\idlelib\rpc.py", line 225, in asynccall
    self.putmessage((seq, request))
  File "C:\Python27\lib\idlelib\rpc.py", line 324, in putmessage
    s = pickle.dumps(message)
  File "C:\Python27\lib\copy_reg.py", line 77, in _reduce_ex
    raise TypeError("a class that defines __slots__ without "
TypeError: a class that defines __slots__ without defining __getstate__ cannot be pickled

I researched the error message but couldn’t find anything I could work with – becasue I don’t want to play with the inner implantation of BeautifulSoup (and honestly I don’t know or understand __slot__ or __getstate__..) I just want to use the functionality.

instead of subclassing I tried returning a beautifulsoup object from the __init__ of the class but __init__ method returns None

Be glad for any help here.

Answer 1

The error is not happening in BeautifulSoup code. Rather, your IDLE is not able to retreive and print the object. Try print str(e) instead.

---

Anyway, subclassing BeautifulSoup in your situation may not be the best idea. Do you really want to inherit all of the parsing methods (like convert_charref, handle_pi or error)? Worse, if you override something that BeautifulSoup uses, it may break in a hard-to-find way.

I don’t know your situation, but I suggest preferring composition over inheritance (i.e. having a BeautifulSoup object in an attribute). You can easily (if in a slightly hacky way) expose specific methods like this:

class Scrape(object):
    def __init__(self, ...):
        self.soup = ...
        ...
        self.find = self.soup.find