Question

When I use simple I/o calls to read a particular file on my system, such as:

f = open('file.ini')
for line in f.readlines():
    print line

I’m getting output such as this:

 H E L L O !  W H Y  A R E  T H E R E  S O  M A N Y  S P A C E S ?

I presume it’s Unicode but I can’t quite figure out how to read it as Unicode / convert it to ascii. Suggestions?

Answer 1

V e r y   r e g u l a r   s p a c e s are usually an indicator that your data is encoded in UTF16 — Usually what you see is that every second byte is a 0 byte. You can confirm this by printing out the actual binary data that you are reading:

f = open('file.ini')
line in f.readline():
print map(ord, line)

If you see output like this:

[..., 68, 0, 65, 0, 76, 0, 76, 0, 79, ...]

Then that’s almost certainly the case.

The trick, then, is to figure out whether it’s the even bytes that are 0s, or the odd bytes. There are two UTF-16 encodings: Big-endian and little-endian, named for the significance of the byte that comes first. If your 0s come before the character that they are associated with, then the file is big-endian, and you can open it like this (Python 3.x):

f = open('file.ini', encoding='utf16be')

In Python 2.x, import the codecs module to do this:

import codecs
f = codecs.open('file.ini', encoding='utf16be')

If the 0s come after, then substitude ‘utf16le’.

(You need to make sure that you decode the file as you’re reading it, or read the entire contents into memory before decoding. You definitly do not want to split lines apart before you decode)

If you’re lucky, then the file was written with a Byte Order Mark at the beginning this character is U+FEFF– if the first two bytes are [254, 255], then the encoding is big-endian, and if [255, 254], then it is little-endian.

If none of those apply, then you might not be looking at UTF-16 data, and you’ll have to do some more research to figure out what encoding you’re looking at.