Question

I have a csv file being read into python, I then save the reader as an array (I guess).

I then compare the csv file results against some Oracle db results:

readerSetSAP = []
readerSAP = csv.reader(StringIO.StringIO(request.POST['sap'].value),dialect=csv.excel)
readerSetSAP.extend(readerSAP)

empsTbl = meta.Session.query(model.Person).all();

Then use a nested loop to compare:

 if i.userid != currEmp[0].strip():
                        updated = True
                        print "userid update"

The problem is, I often have the warning:

eWarning: Unicode unequal comparison failed to convert both arguments to Unicode - interpreting them as being unequal

So my question is:

What is the most robust way of comparing strings of this type in Python?

Answer 1

Your problem here is not ‘a robust way ” to compare strings. A robust way to compare strigns in Python is the equality operator == –
Your problem is that your data is being covnerted to Unicode somewhere, without you being aware of that.

You, and everyone else who writes code, should be aware that text is not ASCII – not in a post 1990 world. Even if all of your application is restricted to English only, and should never run in an internatiol environment, you are bound to find some non-ASCII characters in peoples names, or in words like “resumé”.

Here is a Python console example of when the problem might happen:

>>> "maçã" == u"maçã"
__main__:1: UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal
False

Python’s CSV module do no authomatic conversion, and works with byte strigns (that is – strigns aready converted to some encoding) – which means that that result you are fetching from the DB is in Unicode. Probably your connection is using some default.

To solve that, assuming the data in your database is correctly formatted (and you did not already lost character information during the insertion), is to decode the string read from the CSV file, using an explicit encoding – so that both are in unicode (Python’s internal encoding agnostic) string format –

>>> "maçã".decode("utf-8") == u"maçã"
True

So, you do use the “decode” method on the string read form the CSV file in order to have a proepr conversion, before comparing it. If you are on Windows, use the “cp1251” for decoding., In any other mainstream (application) O.S. it should be “utf-8”.

I’d advise reading of this piece – it is rather useful:
http://www.joelonsoftware.com/articles/Unicode.html