Question

I built a django site last year that utilises both a dashboard and an API for a client.

They are, on occasion, putting unicode information (usually via a Microsoft keyboard and a single quote character!) into the database.

It’s fine to change this one instance for everything, but what I constantly get is something like this error when a new character is added that I haven’t “converted”:

UnicodeDecodeError at /xx/xxxxx/api/xxx.json

‘ascii’ codec can’t decode byte 0xeb in position 0: ordinal not in range(128)

The issue is actually that I need to be able to convert this unicode (from the model) into HTML.

# if a char breaks the system, replace it here (duplicate line)
text = unicode(str(text).replace('\xa3', '£'))

I duplicate this line here, but it just breaks otherwise.

Tearing my hair out because I know this is straight forward and I’m doing something remarkably silly somewhere.

Have searched elsewhere and realised that while my issue is not new, I can’t find the answer elsewhere.

Answer 1

I assume that text is unicode (which seems a safe assumption, as \xa3 is the unicode for the £ character).

I’m not sure why you need to encode it at all, seeing as the text will be converted to utf-8 on output in the template, and all browsers are perfectly capable of displaying that. There is likely another point further down the line where something (probably your code, unfortunately) is assuming ASCII, and the implicit conversion is breaking things.

In that case, you could just do this:

text = text.encode('ascii', 'xmlcharrefreplace')

which converts the non-ASCII characters into HTML/XML entities like £.