Question

I am struggling with the following issue: I have an XML string that contains the following tag and I want to convert this, using cElementTree, to a valid XML document:

<tag>#55296;#57136;#55296;#57149;#55296;#57139;#55296;#57136;#55296;#57151;#55296;
#57154;#55296;#57136;</tag>

but each # sign is preceded by a & sign and hence the output looks like: ��������������

This is a unicode string and the encoding is UTF-8. I want to discard these numeric character references because they are not legal XML in a valid XML document (see Parser error using Perl XML::DOM module, "reference to invalid character number")

I have tried different regular expression to match these numeric character references. For example, I have tried the following (Python) regex:

RE_NUMERIC_CHARACTER = re.compile('&#[\d{1,5}]+;')

This does work in regular python session but as soon as I use the same regex in my code then it doesn’t work, presumably because those numeric characters have been interpreted (and are shown as boxes or question marks).

I have also tried the unescape function from http://effbot.org/zone/re-sub.htm but that does not work either.

Thus: how can I match, using a regular expression in Python, these numeric character references and create a valid XML document?

Answer 1

Eurgh. You’ve got surrogates (UTF-16 code units in the range D800-DFFF), which some fool has incorrectly encoded individually instead of using a pair of code units for a single character. It would be ideal to replace this mess with what it should look like:

<tag>&#66352;&#66365;&#66355;&#66352;&#66367;&#66370;&#66352;</tag>

Or, just as valid, in literal characters (if you’ve got a font that can display the Gothic alphabet):

<tag></tag>

Usually, it would be best to do replacement operations like this on parsed text nodes, to avoid messing up non-character-reference sequences in other places like comments or PIs. However of course that’s not possible in this case since this isn’t really XML at all. You could try to fix it up with a crude regex, though it would be better to find out where the invalid input is coming from and kick the person responsible until they fix it.

>>> def lenient_deccharref(m):
...    return unichr(int(m.group(1)))
...
>>> tag= '<tag>&#55296;&#57136;&#55296;&#57149;&#55296;&#57139;&#55296;&#57136;&#55296;&#57151;&#55296;&#57154;&#55296;&#57136;</tag>'
>>> re.sub('&#(\d+);', lenient_deccharref, tag).encode('utf-8')
'<tag>\xf0\x90\x8c\xb0\xf0\x90\x8c\xbd\xf0\x90\x8c\xb3\xf0\x90\x8c\xb0\xf0\x90\x8c\xbf\xf0\x90\x8d\x82\xf0\x90\x8c\xb0</tag>'

This is the correct UTF-8 encoding of . The utf-8 codec allows you to encode a sequence of surrogates to correct UTF-8 even on a wide-Unicode platform where the surrogates should not have appeared in the string in the first place.

>>> _.decode('utf-8')
u'<tag>\U00010330\U0001033d\U00010333\U00010330\U0001033f\U00010342\U00010330</tag>'