Question

I’m using python to read a text file with the segment below

(can’t post a screenshot since i’m a noob) but this is what is looks like in notepad++:

NULSOHSOHNULNULNULSUBMesssage-ID:

error:

Traceback (most recent call last):
  File "<pyshell#3>", line 1, in <module>
    print(f.readline())
  File "C:\Python32\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x8f in position 7673: character maps to <undefined>

Opening the file as binary:

f = open('file.txt','rb')
f.readline()

gives me the text as binary

b’\x00\x01\x01\x00\x00\x00\x1a\xb7Message-ID:

but how do I get the text as ascii ? And whats the easiest/pythonic way of handling this ?

Answer 1

The problem is with “byte 0x8f in position 7673”, not with “byte 0x00 in position 1”. I.e., your NUL is not the problem. If you look at the cp-1252 codepage on wikipedia, you can see that 0x8f has no corresponding character.

The larger issue is that your file is not in a single encoding: it appears to be a mix of binary framing of text segments. What you really need to do is figure out the format of this file and parse it into binary pieces (or perhaps some richer data structure, like a tuple, list, dict, object, etc), then decode the text pieces into unicode if you need to process further.