Question

I have recently upgrade BeautifulSoup from version 3.0 to version 4.1 on a Windows machine.

I am now getting a strange error:

File "C:\path\to\myscript.py", line 23
0, in soupify
    return BeautifulSoup(html, convertEntities=BeautifulSoup.HTML_ENTITIES)
AttributeError: type object 'BeautifulSoup' has no attribute 'HTML_ENTITIES'

Here is the snippet of code that causes the exception to be thrown:

def soupify(html):
    return BeautifulSoup(html, convertEntities=BeautifulSoup.HTML_ENTITIES)

The doc for BS does not mention how the constructor signature has changed fro v3 to v4. How may I fix this?

Answer 1

An incoming HTML or XML entity is always converted into the
corresponding Unicode character. Beautiful Soup 3 had a number of
overlapping ways of dealing with entities, which have been removed.
The BeautifulSoup constructor no longer recognizes the smartQuotesTo
or convertEntities arguments. (Unicode, Dammit still has
smart_quotes_to, but its default is now to turn smart quotes into
Unicode.)

If you want to turn those Unicode characters back into HTML entities
on output, rather than turning them into UTF-8 characters, you need to
use an output formatter.

Source: http://www.crummy.com/software/BeautifulSoup/bs4/doc/#entities