Question

I’m trying BeautifulSoup for parsing html files which is encoded in UTF-8. But unfortunately, this html file contains a few characters which are non-utf-8 character, hence not showed correctly. But this is OK for me since I could just simply skip these characters.

The problems is, even if I directly specify the encodingFrom as utf-8:

soup = BeautifulSoup (html,fromEncoding='utf-8')

It turns out the soup.originalEncoding is set to default windows-1252 automatically.

print soup.originalEncoding
windows-1252

I referred to the BeautifulSoup Documents and it’s written like:

Beautiful Soup tries the following encodings, in order of priority, to turn your document into Unicode:

 - An encoding you pass in as the fromEncoding argument to the soup
   constructor.
 - An encoding discovered in the document itself
 - An encoding sniffed by looking at the first few bytes of the file. If
   an encoding is detected at this stage, it will be one of the UTF-*
   encodings, EBCDIC, or ASCII.
 - An encoding sniffed by the chardet library, if you have it installed.
 - UTF-8
 - Windows-1252

It seems it should use fromEncoding I specified instead of falling to the last one in the list.

Here is the original html I’m parsing for your reference.

Answer 1

If you know what the encoding of file will be, try decoding your string before passing it to BeautifulSoup and explicitly ignore non-utf8 characters.

unicode_html = myfile.read().decode('utf-8', 'ignore')
soup = BeautifulSoup (unicode_html)