Question

I’m having some brain failure in understanding reading and writing text to a file (Python 2.4).

# The string, which has an a-acute in it. ss = u'Capit\xe1n' ss8 = ss.encode('utf8') repr(ss), repr(ss8) 

(‘u’Capit\xe1n”, ”Capit\xc3\xa1n”)

print ss, ss8 print >> open('f1','w'), ss8  >>> file('f1').read() 'Capit\xc3\xa1n\n' 

So I type in Capit\xc3\xa1n into my favorite editor, in file f2.

Then:

>>> open('f1').read() 'Capit\xc3\xa1n\n' >>> open('f2').read() 'Capit\\xc3\\xa1n\n' >>> open('f1').read().decode('utf8') u'Capit\xe1n\n' >>> open('f2').read().decode('utf8') u'Capit\\xc3\\xa1n\n' 

What am I not understanding here? Clearly there is some vital bit of magic (or good sense) that I’m missing. What does one type into text files to get proper conversions?

What I’m truly failing to grok here, is what the point of the UTF-8 representation is, if you can’t actually get Python to recognize it, when it comes from outside. Maybe I should just JSON dump the string, and use that instead, since that has an asciiable representation! More to the point, is there an ASCII representation of this Unicode object that Python will recognize and decode, when coming in from a file? If so, how do I get it?

>>> print simplejson.dumps(ss) ''Capit\u00e1n'' >>> print >> file('f3','w'), simplejson.dumps(ss) >>> simplejson.load(open('f3')) u'Capit\xe1n' 

Answer 1

In the notation u'Capit\xe1n\n' (should be just 'Capit\xe1n\n' in 3.x, and must be in 3.0 and 3.1), the \xe1 represents just one character. \x is an escape sequence, indicating that e1 is in hexadecimal.

Writing Capit\xc3\xa1n into the file in a text editor means that it actually contains \xc3\xa1. Those are 8 bytes and the code reads them all. We can see this by displaying the result:

# Python 3.x - reading the file as bytes rather than text, # to ensure we see the raw data >>> open('f2', 'rb').read() b'Capit\\xc3\\xa1n\n'  # Python 2.x >>> open('f2').read() 'Capit\\xc3\\xa1n\n' 

Instead, just input characters like á in the editor, which should then handle the conversion to UTF-8 and save it.

In 2.x, a string that actually contains these backslash-escape sequences can be decoded using the string_escape codec:

# Python 2.x >>> print 'Capit\\xc3\\xa1n\n'.decode('string_escape') Capitán 

The result is a str that is encoded in UTF-8 where the accented character is represented by the two bytes that were written \\xc3\\xa1 in the original string. To get a unicode result, decode again with UTF-8.

In 3.x, the string_escape codec is replaced with unicode_escape, and it is strictly enforced that we can only encode from a str to bytes, and decode from bytes to str. unicode_escape needs to start with a bytes in order to process the escape sequences (the other way around, it adds them); and then it will treat the resulting \xc3 and \xa1 as character escapes rather than byte escapes. As a result, we have to do a bit more work:

# Python 3.x >>> 'Capit\\xc3\\xa1n\n'.encode('ascii').decode('unicode_escape').encode('latin-1').decode('utf-8') 'Capitán\n'