Question

I crawled a set of sites and extracted different strings with unicode encoded characters such as ‘Best places to eat in D\xfcsseldorf’. I have them stored as showed in a PostgreSQL database. When I retrieve strings that the mentioned earlier from Database and do:

name = string_retrieved_from_database
print name

outputs as unicode u’Best places to eat in D\xfcsseldorf’. I want to display the string as it should be: ‘Best places to eat in Düsseldorf’. How can I do that.

Answer 1

Are you sure you get output when you print the variable, instead of just displaying it interactively? You should never get the u'...' display when using print:

>>> x = b"Best places to eat in D\xfcsseldorf"
>>> x.decode('latin-1')
u'Best places to eat in D\xfcsseldorf'
>>> print x.decode('latin-1')
Best places to eat in Düsseldorf

If you’re getting the backslash and so forth in the actual string, then it’s possible something went wrong at the encoding stage (e.g., literal backslashes were written into the text). In that case you may want to look at the “unicode-escape” codec:

>>> x = b"Best places to eat in D\\xfcsseldorf"
>>> print x
Best places to eat in D\xfcsseldorf
>>> print x.decode('unicode-escape')
Best places to eat in Düsseldorf