Question

I have a list with different strings. Sometimes they are in cp1251, ASCII or something else. I need to process them (convert to Unicode), because I got an error (UncicodeDecodeError), especially when I tried to dump this data to JSON.

How can I do this?

Answer 1

You can use chardet to detect the encoding of a string, so one way to convert a list of them to unicode (in Python 2.x) would be:

import chardet

def unicodify(seq, min_confidence=0.5):
    result = []
    for text in seq:
        guess = chardet.detect(text)
        if guess["confidence"] < min_confidence:
            # chardet isn't confident enough in its guess, so:
            raise UnicodeDecodeError
        decoded = text.decode(guess["encoding"])
        result.append(decoded)
    return result

… which you’d use like this:

>>> unicodify(["¿qué?", "什么？", "what?"])
[u'\xbfqu\xe9?', u'\u4ec0\u4e48\uff1f', u'what?']

CAVEAT: Solutions like chardet should only be used as a last resort (for instance, when repairing a dataset that’s corrupt because of past mistakes). It’s far too fragile to be relied on in production code; instead, as @bames53 points out in the comments to this answer, you should fix the code that corrupted the data in the first place.