Question

I’ve create little class for parsing websites.

There’s URLError exception:

def visit(self, url, referer=None, data=None):

    (...)

    # Return BeautifulSoup instance.
    try:
        return BeautifulSoup(self.opener.open(self.request))
    # URLError.
    except urllib.error.URLError as error:
        return error

Everything works okay. But I’m in need to create a wrapper of this function.

def get_links(self, *args, **kwargs):

    # Get links with BeautifulSoup.
    self.links = self.visit(*args, **kwargs).find_all('a')

Get_links function also works well until there is URLError (403, 404, whatever…). How can I solve this problem? Is there something as inheritance exceptions?

Answer 1

Your visit() function catches exception and returns you a URLError object, on which you’re calling find_all(), which it doesn’t have.

Something in lines of:

self.links = self.visit(*args, **kwargs)
if not isinstance(self.links, urllib.error.URLError):
    self.links = self.links.find_all('a')
else:
    # Do something with an HTTP Error

Should give you an idea of a flow. You can’t catch that exception in your outer get_links() because it’s already caught by visit() and is simply returned.

If you want to catch it in get_links(), change

return error

to

raise error

in your visit() method, although then you’ll be throwing the exception you just caught again, I’m not sure whether this is the behavior you want.