Question

python 3.x

>>> a = input()
hope
>>> a
'hope'
>>> b = input()
håpe
>>> b
'håpe'
>>> c = input()

start typing hå... delete using backspace... and change to hope

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf8' codec can't decode byte 0xc3 in position 1: invalid continuation byte
>>> 

The situation is not terrible, I am working around it, but find it strange that when deleting, the bytes get messed up. Has anyone else experienced this?

the terminal history shows that it thought that I entered h?ope

any ideas?

in the script that is using this, I do import readline to give command line history.

Answer 1

It looks like backspace is deleting the last byte instead of the last character. What ends up happening is that you type:

68 c3 a5
h |  å

68 c3 6f 70 65
h |  |o |p |e

In UTF-8, a byte with the first bit set (c3) means that the next byte must have its first bit set as well (see Wikipedia’s description).

Make sure that your terminal emulator and readline understand you’re using UTF-8.