Question

☺:

>>> bytes('☺','ibm437')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.3/encodings/cp437.py", line 12, in encode
    return codecs.charmap_encode(input,errors,encoding_map)
UnicodeEncodeError: 'charmap' codec can't encode character '\u263a' in position 0: character maps to <undefined>

As opposed to é, which works:

>>> bytes('é','ibm437')
b'\x82'

I expect ☺ to bring me back b'\x01'. How can I make this the case?

An image of Code Page 437.

Answer 1

IBM-437 is somewhat special in that it is not only a codepage (i.e. defines what should happen for byte values 128-255), but also redefines some of the ASCII control characters, but only in some contexts. Python maps those problematic codepoints to control characters, and not to the visible characters they were displayed as in some contexts.

To convert, you can use the following helper method:

ibm437_visible = lambda byt: byt.decode('ibm437').translate({
    0x01: "\u263A", 0x02: "\u263B", 0x03: "\u2665", 0x04: "\u2666",
    0x05: "\u2663", 0x06: "\u2660", 0x07: "\u2022", 0x08: "\u25D8",
    0x09: "\u25CB", 0x0a: "\u25D9", 0x0b: "\u2642", 0x0c: "\u2640",
    0x0d: "\u266A", 0x0e: "\u266B", 0x0f: "\u263C", 0x10: "\u25BA",
    0x11: "\u25C4", 0x12: "\u2195", 0x13: "\u203C", 0x14: "\u00B6",
    0x15: "\u00A7", 0x16: "\u25AC", 0x17: "\u21A8", 0x18: "\u2191", 
    0x19: "\u2193", 0x1a: "\u2192", 0x1b: "\u2190", 0x1c: "\u221F",
    0x1d: "\u2194", 0x1e: "\u25B2", 0x1f: "\u25BC", 0x7f: "\u2302",
})
assert ibm437_visible(b'\x01') == '☺'