Question

I’ve a programm that gets an argument from the shell. This argument will be the query used in a search operation.

If I pass in English words (i.e. no accents, etc.), it works fine. Nevertheless, if I pass in, namely, ‘café’, I get ‘cafú’ (print sys.argv[1] results in cafÚ instead of café).

I thought I could solve the problem by converting it into a Unicode object, but I was wrong.

Q = unicode(sys.argv[1], encoding=sys.stdin.encoding)

I still get ‘cafÚ’!! I’m going crazy…

Answer 1

I bet you’re on Windows, right?

>>> a = "café"
>>> a
'caf\x82'
>>> print a
café
>>> a.decode("cp850")                      # DOS codepage 850 --> Unicode
u'caf\xe9'
>>> a.decode("cp850").encode("cp1252")     # DOS 850 --> Unicode --> Windows 1252
'caf\xe9'                                  # identical to Unicode codepoint
>>> print a.decode("cp850").encode("cp1252") # Display a cp1252 string in cp850
cafÚ

Use encoding="cp1252" instead, then it should work.

Explanation: (with some guesswork)

• cmd windows use cp850 as their default codepage. This is evident from the second line in my session above, 0x82 is é in cp850.
• It appears that Python programs started under Windows use cp1252 as their standard encoding, shown by the last line of the session above: é is 0xe9 in cp1252 (like in Unicode).
• This is also evident when you write this string to a file (which by default uses cp1252):
  If I do f.write(a), I get caf, as the contents of my file because , is 0x82 in cp1252).
  If I do f.write(a.decode("cp850").encode("cp1252")), I get café.

Moral: Find out the correct encodings in your environment, convert everything to Unicode as soon as possible, work with it, then convert back to the encoding you need. If you’re outputting into an interactive window, use cp850, if you’re outputting into a file, use cp1252.

Or switch to Python 3 which makes all of this much easier.