Question

I currently have serious problems with coding/encoding under Linux (Ubuntu). I never needed to deal with that before, so I don’t have any idea why this actually doesn’t work!

I’m parsing *.desktop files from /usr/share/applications/ and extracting information which is shown in the Web browser via a HTTPServer. I’m using jinja2 for templating.

First, I received UnicodeDecodeError at the call to jinja2.Template.render() which said that

utf-8 cannot decode character XXX at position YY [...]

So I have made all values that come from my appfind-module (which parses the *.desktop files) returning only unicode-strings.

The problem at this place was solved so far, but at some point I am writing a string returned by a function to the BaseHTTPServer.BaseHTTTPRequestHandler.wfile slot, and I can’t get this error fixed, no matter what encoding I use.

At this point, the string that is written to wfile comes from jinja2.Template.render() which, afaik, returns a unicode object.

The bizarre part is, that it is working on my Ubuntu 12.04 LTS but not on my friend’s Ubuntu 11.04 LTS. However, that might not be the reason. He has a lot more applications and maybe they do use encodings in their *.desktop files that raise the error.

However, I properly checked for the encoding in the *.desktop files:

data = dict(parser.items('Desktop Entry'))

try:
    encoding = data.get('encoding', 'utf-8')
    result = {
        'name':       data['name'].decode(encoding),
        'exec':       DKENTRY_EXECREPL.sub('', data['exec']).decode(encoding),
        'type':       data['type'].decode(encoding),
        'version':    float(data.get('version', 1.0)),
        'encoding':   encoding,
        'comment':    data.get('comment', '').decode(encoding) or None,
        'categories': _filter_bool(data.get('categories', '').
                                        decode(encoding).split(';')),
        'mimetypes':  _filter_bool(data.get('mimetype', '').
                                        decode(encoding).split(';')),
    }

# ...

Can someone please enlighten me about how I can fix this error? I have read on an answer on SO that I should use unicode() always, but that would be so much pain to implemented, and I don’t think it would fix the problem when writing to wfile?

Thanks,
Niklas

Answer 1

You need to take a disciplined approach to your byte strings and Unicode strings. This explains it all: Pragmatic Unicode, or, How Do I Stop the Pain?