Question

I have an Excel spreadsheet that I’m reading in that contains some £ signs.

When I try to read it in using the xlrd module, I get the following error:

x = table.cell_value(row, col)
x = x.decode("ISO-8859-1")
UnicodeEncodeError: 'ascii' codec can't encode character u'\xa3' in position 0: ordinal not in range(128)

If I rewrite this to x.encode(‘utf-8’) it stops throwing an error, but unfortunately when I then write the data out somewhere else (as latin-1), the £ signs have all become garbled.

How can I fix this, and read the £ signs in correctly?

— UPDATE —

Some kind readers have suggested that I don’t need to decode it at all, or that I can just encode it to Latin-1 when I need to. The problem with this is that I need to write the data to a CSV file eventually, and it seems to object to the raw strings.

If I don’t encode or decode the data at all, then this happens (after I’ve added the string to an array called items):

for item in items:
    #item = [x.encode('latin-1') for x in item]
    cleancsv.writerow(item)
File "clean_up_barnet.py", line 104, in <module>
 cleancsv.writerow(item)
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2022' in position 43: ordinal not in range(128)

I get the same error even if I uncomment the Latin-1 line.

Answer 1

Your code snippet says x.decode, but you’re getting an encode error — meaning x is Unicode already, so, to “decode” it, it must be first turned into a string of bytes (and that’s where the default codec ansi comes up and fails). In your text then you say “if I rewrite ot to x.encode“… which seems to imply that you do know x is Unicode.

So what it IS you’re doing — and what it is you mean to be doing — encoding a unicode x to get a coded string of bytes, or decoding a string of bytes into a unicode object?

I find it unfortunate that you can call encode on a byte string, and decode on a unicode object, because I find it seems to lead users to nothing but confusion… but at least in this case you seem to manage to propagate the confusion (at least to me;-).

If, as it seems, x is unicode, then you never want to “decode” it — you may want to encode it to get a byte string with a certain codec, e.g. latin-1, if that’s what you need for some kind of I/O purposes (for your own internal program use I recommend sticking with unicode all the time — only encode/decode if and when you absolutely need, or receive, coded byte strings for input / output purposes).