Home/ Questions/Q 612569
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-13T17:54:28+00:00

This is somehow related to my question here . I process tons of texts

  • 0

This is somehow related to my question here.

I process tons of texts (in HTML and XML mainly) fetched via HTTP. I’m looking for a library in python that can do smart encoding detection based on different strategies and convert texts to unicode using best possible character encoding guess.

I found that chardet does auto-detection extremely well. However auto-detecting everything is the problem because it is SLOW and very much against all standards. As per chardet FAQ I don’t want to screw the standards.

From the same FAQ here is the list of places where I want to look for encoding:

  • charset parameter in HTTP Content-type header.
  • <meta http-equiv="content-type"> element in
    the <head> of a web page for HTML
    documents.
  • encoding attribute in the XML prolog for XML
    documents.
  • Auto-detect the character encoding as a last resort.

Basically I want to be able to look in all those place and also deal with conflicting information automatically.

Is there such library out there or do I need to write it myself?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    BeautifulSoup (the html parser) incorporates a class called UnicodeDammit that does just that. Have a look and see if you like it.

    • 0