Question

I’m working on extracting data from a SQL Server database with a latin_1 character set into a Greenplum/postgres database with a utf-8 character set. I’m trying to convert the string values immediately before insert, but when I do this:

row=[i.decode('latin_1') for i in row]
row=[i.encode('utf-8') for i in row]

I get an error stating that decode is not a member of type int. That makes sense in that there are integer values coming in. But there are also strings. In other posts of this kind I’ve read, the answer was always immediately and resoundingly ‘you should always know what type is coming over’. In many respects I do, since it’s a static query, but it seems awfully klunky, and honestly unmaintainable, to define a set of values for i in which I want to do the conversion for each and every query I write. It would seem type testing would be the clean, encapsulable, and reusable answer here, no?

Any suggestions?

Answer 1

I’d use a small function like this:

def convert(s):
    try:
        return s.decode('latin-1').encode('utf8')
    except AttributeError:
        return s

and then

row = map(convert, row) # or a compr if you prefer that

The advantage is that it also handles types other than int automatically.