Question

I have a script that needs to determine the charset before being read by lxml.HTML() for parsing. I will assume ISO-8859-1(that’s the normal assumed charset for this right?) if it can’t be found and search the html for the meta tag with the charset attribute. However I’m not sure the best way to do that. I could try to create an etree with lxml, but I don’t want to read the whole file since I may run into encoding problems. However, if I don’t read the whole file I can’t build an etree since some tags will not have been closed.

Should I just find the meta tag with some fancy string subscripting and break out of the loop once it’s found or a certain number of lines have been read? Maybe use a low level HTML parser, eg html.parser? Using python3 btw, thanks.

Answer 1

You should first try to extract encoding from HTTP headers. If it is not present there, you should parse it with the lxml. This might be tricky since lxml throws parse errors if charset does not match. A work-around would be decoding and encoding the data ignoring the unknown characters.

html_data=html_data.decode("UTF-8","ignore")
html_data=html_data.encode("UTF-8","ignore")

After this, you can parse by invoking the lxml.HTML() command with utf-8 encoding.
This way, you’ll be able to find the correct encoding defined in the HTML headers.

After finding the encoding, you’ll have to re-parse the HTML document with proper encoding.

Unfortunately, sometimes you might not find character encoding even in the HTML headers. I’d suggest you using the chardet module to find the proper encoding only after these steps fail.