Question

I have been studying unicode and its Python implementation now for two days, and I think I’m getting a glimpse of what it is about. Just to get confident, I’m asking if my assumptions for my current problems are correct.

In Django, forms give me unicode strings which I suspect to be “broken”. Unicode strings in Python should be encoded in UTF-8, is that right? After entering the string “fähre” into a text field, the browser sends the string “f%c3%a4hre” in the POST request (checked via wireshark). When I retrieve the value via form.cleaned_data, I’m getting the string u’f\xa4hre’ (note it is a unicode string), though. As far as I understand that, that is ISO-8859-1-encoded unicode string, which is incorrect. The correct string should be u’f\xc3\xa4hre’, which would be a UTF-8-encoded unicode string. Is that a Django bug or is there something wrong with my understanding of it?
To fix the issue, I wrote a function to apply it to any text input from Django forms:

def fix_broken_unicode(s):
    return unicode(s.encode(u'utf-8'), u'iso-8859-1')

which does

>>> fix_broken_unicode(u'f\xa4hre')
u'f\xc3\xa4hre'

That doesn’t seem very elegant to me, but setting Django’s settings.DEFAULT_CHARSET to ‘utf-8′ didn’t help, nor did anything else. I am trying to work with unicode throughout the whole application so I won’t get any weird errors later on, but it obviously does not suffice to mark all strings with u’…’.

Edit: Considering the answers from Dirk and sth, I will now save the strings to the database as they are. The real problem was that I was trying to urlencode these kinds of strings to use them as input for the Twitter API etc. In GET or POST requests, though, UTF-8 encoding is obviously expected which the standard urllib.urlencode() function does not process correctly (throws exceptions). Take a look at my solution in the pastebin and feel free to comment on it also.

Answer 1

u'f\xa4hre'is a unicode string, not encoded as anything. The unicode codepoint 0xa4 is the character ä. It is not really important that ä would also be encoded as byte 0xa4 in ISO-8859-1.

The unicode string can contain any unicode characters without encoding them in some way. For example 轮渡 would be represented as u'\u8f6e\u6e21', which are simply two unicode codepoints. The UTF-8 encoding would be the much longer '\xe8\xbd\xae\xe6\xb8\xa1'.

So there is no need to fix the encoding, you are just seeing the internal representation of the unicode string.