Question

The following code causes the well-known “UnicodeDecodeError: ‘ascii’ codec can’t decode” error:

import xml.sax
import io
parser = xml.sax.make_parser()
parser.parse(io.StringIO(u'<a>é</a>'))

While

import xml.sax
parser = xml.sax.make_parser()
parser.parse(open('foo'))

works (the content of file “foo” is <a>é</a>).

I need to parse an XML string in my case, not a file.

Is there any solution to my problem? Thanks.

Answer 1

A file contains bytes, and must have some encoding to store Unicode characters, so use a BytesIO object instead:

#coding: utf8
import xml.sax 
import io 
parser = xml.sax.make_parser() 
parser.parse(io.BytesIO(u'<a>é</a>'.encode('utf8')))

Note: #coding: utf8 specifies the encoding of the source file; .encode('utf8') specifies the encoding of the Unicode string to be stored in the BytesIO object. Technically using a non-Unicode string:

#coding: utf8
parser.parse(io.BytesIO('<a>é</a>'))

will work as well, since byte strings will be in the source file encoding already, but it makes the intent clearer. The source file and BytesIO encoding could be different.