Question

I’m currently persisting filenames in a sqlite database for my own purposes. Whenever I try to insert a file that has a special character (like é etc.), it throws the following error:

pysqlite2.dbapi2.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

When I do “switch my application over to Unicode strings” by wrapping the value sent to pysqlite with the unicode method like: unicode(filename), it throws this error:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 66: ordinal not in range(128)

Is there something I can do to get rid of this? Modifying all of my files to conform isn’t an option.

UPDATE
If I decode the text via filename.decode("utf-8"), I’m still getting the ProgrammingError above.

My actual code looks like this:

cursor.execute("select * from musiclibrary where absolutepath = ?;",
    [filename.decode("utf-8")])

What should my code here look like?

Answer 1

You need to specify the encoding of filename for conversion to Unicode, for example: filename.decode('utf-8'). Just using unicode(...) picks the console encoding, which is often unreliable (and often ascii).