Question

On a Python driven web app using a sqlite datastore I had this error:

Could not decode to UTF-8 column
‘name’ with text ‘300µL-10-10’

Reading here it looks like I need to switch my text-factory to str and get bytestrings but when I do this my html output looks like this:

300�L-10-10

I do have my content-type set as:

<meta http-equiv="content-type" content="text/html; charset=utf-8" />

Answer 1

Unfortunately, the data in your datastore is not encoded as UTF-8; instead, it’s probably either latin-1 or cp1252. To decode it automatically, try setting Connection.text_factory to your own function:

def convert_string(s):
    try:
        u = s.decode("utf-8")
    except UnicodeDecodeError:
        u = s.decode("cp1252")
    return u

conn.text_factory = convert_string