Question

I’m falling the unicode hell.

My environment in on unix, python 2.7.3

LC_CTYPE=zh_TW.UTF-8
LANG=en_US.UTF-8

I’m trying to dump hex encoded data in human readable format, here is simplified code

#! /usr/bin/env python
# encoding:utf-8
import sys

s=u"readable\n"  # previous result keep in unicode string
s2="fb is not \xfb"  # data read from binary file
s += s2

print s   # method 1
print s.encode('utf-8')  # method 2
print s.encode('utf-8','ignore')  # method 3
print s.decode('iso8859-1')  # method 4

# method 1-4 display following error message
#UnicodeDecodeError: 'ascii' codec can't decode byte 0xfb 
# in position 0: ordinal not in range(128)

f = open('out.txt','wb')
f.write(s)

I just want to print out the 0xfb.

I should describe more here. The key is ‘s += s2’.
Where s will keep my previous decoded string.
And the s2 is next string which should append into s.

If I modified as following, it occurs on write file.

s=u"readable\n"
s2="fb is not \xfb"
s += s2.decode('cp437')
print s
f=open('out.txt','wb')
f.write(s)
# UnicodeEncodeError: 'ascii' codec can't encode character
# u'\u221a' in position 1: ordinal not in range(128)

I wish the result of out.txt is

readable
fb is not \xfb

or

readable
fb is not 0xfb

[Solution]

#! /usr/bin/env python
# encoding:utf-8
import sys
import binascii

def fmtstr(s):
    r = ''
    for c in s:
        if ord(c) > 128:
            r = ''.join([r, "\\x"+binascii.hexlify(c)])
        else:
            r = ''.join([r, c])
    return r

s=u"readable"
s2="fb is not \xfb"
s += fmtstr(s2)
print s
f=open('out.txt','wb')
f.write(s)

Answer 1

I strongly suspect that your code is actually erroring out on the previous line: the s += s2 one. s2 is just a series of bytes, which can’t be arbitrarily tacked on to a unicode object (which is instead a series of code points).

If you had intended the ‘\xfb’ to represent U+FB, LATIN SMALL LETTER U WITH CIRCUMFLEX, it would have been better to assign it like this instead:

s2 = u"\u00fb"

But you said that you just want to print out \xHH codes for control characters. If you just want it to be something humans can understand which still makes it apparent that special characters are in a string, then repr may be enough. First, don’t have s be a unicode object, because you’re treating your strings here as a series of bytes, not a series of code points.

s = s.encode('utf-8')
s += s2

print repr(s)

Finally, if you don’t want the extra quotes on the outside that repr adds, for nice pretty printing or whatever, there’s not a simple builtin way to do that in Python (that I know of). I’ve used something like this before:

import re
controlchars_re = re.compile(r'[\x00-\x31\x7f-\xff]')

def _show_control_chars(match):
    txt = repr(match.group(0))
    return txt[1:-1]

def escape_special_characters(s):
    return controlchars_re.sub(_show_control_chars, s.replace('\\', '\\\\'))

You can pretty easily tweak the controlchars_re regex to define which characters you care about escaping.