Question

I am getting a UnicodeDecodeError: ‘utf8’ codec can’t decode bytes… invalid start byte.

I suspect it has to do with one of the values in my dictionary. To access all fields and put them into a dict, I use:

        mydictionary = {x:y for x,y in zip(column, values)}

What could I change to make it so that I can guarantee that the values could be converted into some way that is utf8 compliant or to avoid this error?

column contains all column headers… values contains a tuple with all values that correspond to the column

i.e.
column = (‘NAME’, HOBBY’)
values = (‘George’, ‘Basketball’)

The issue I am having is that somewhere in values, there is something going on thats like:
values = (‘-insert strange utf8 noncompliant character-George’, ‘Basketball’)

Answer 1

If you don’t care about the exact content of the bad values, you can simply tell the UTF-8 codec to ignore errors,

import codecs
codec = codecs.lookup('utf-8')
mydictionary = {codec.decode(x, 'ignore'): codec.decode(y, 'ignore') for x,y in zip(column, values)}

Alternatively, replacing 'ignore' with 'replace' will cause the codec to replace any misformed characters with the Unicode “replacement character” code point (U+FFFD). If you are only concerned about misformed strings in values, you can obvious omit the decode call on the key.