Question

I am decoding a string “öçÇşŞükrÜ” to some other charset.

I encountered some interesting problem and want to learn how I can solve it.

>>> "öçÇşŞükrÜ".decode("utf-8")
u'\xf6\xe7\xc7\u015f\u015e\xfckr\xdc'
>>> "öçÇşŞükrÜ".decode("utf-8").encode("utf-8")
'\xc3\xb6\xc3\xa7\xc3\x87\xc5\x9f\xc5\x9e\xc3\xbckr\xc3\x9c'


>>> import chardet
>>> chardet.detect('\xc3\xb6\xc3\xa7\xc3\x87\xc5\x9f\xc5\x9e\xc3\xbckr\xc3\x9c')
{'confidence': 0.99, 'encoding': 'utf-8'}
>>> chardet.detect("öçÇşŞükrÜ")
{'confidence': 0.99, 'encoding': 'utf-8'}

What I do not understand and can not find is how to retrieve the original string “öçÇşŞükrÜ” with an decode and encode process with Python ?

Answer 1

I am decoding a string “öçÇşŞükrÜ” to some other charset.

No, you’re decoding an encoded string into an in-memory Unicode representation. Sorry if that sounds harsh, but IMHO it’s really important to get the basics right with Unicode and not just fumble around until something works. EDIT: The resources in the comments, https://www.youtube.com/watch?v=sgHbC6udIqc and http://www.joelonsoftware.com/articles/Unicode.html are also essential. Read and watch them. What you need to know is that nothing on disk, or in your terminal is Unicode. Everything is encoded, using one encoding or another.

If you are using Linux, you’re lucky! (Us Windows devs… well, world of pain, etc.) Your console will most likely be UTF-8, so doing

"öçÇşŞükrÜ".decode("UTF-8")

will probably make sense.

Did you try

print "öçÇşŞükrÜ".decode("UTF-8").encode("UTF-8")

(that is, with print applied)? You should then get your original öçÇşŞükrÜ back.