Question

I am having some problems with accents.

I did a python script that are getting the word “refeição” from some input (IMAP fetch), this word is in Portuguese and I need convert it to be human readable. After decode, it should appear like “refeição” but I am not getting this result…

>>> print a 
refeição
>>> ENCODING = locale.getpreferredencoding()
>>> print ENCODING
UTF-8
>>> print a.encode(ENCODING)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 5: ordinal not in range(128)
>>> a.decode('utf-8')
u'refei\xe7\xe3o'
>>> print a.decode('utf-8')
refeição

Updated:

root@ticuna:/etc/scripts# locale
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Also, theses words are inserted in a mysql database and the “unreadable” characters are showing in the same way that is in terminal.
The table collation is utf8_general_ci

Answer 1

As work around, I am removing all accents.

Here is the code that I used:

def remove_accents(s):
   return ''.join((c for c in unicodedata.normalize('NFD', s.decode('utf-8')) if unicodedata.category(c) != 'Mn'))

Based in this answer:
What is the best way to remove accents in a Python unicode string?