Question

I am working on a program (Python 2.7) that reads xls files (in MHTML format). One of the problems I have is that files contain symbols/characters that are not ascii. My initial solution was to read the files in using unicode

Here is how I am reading in a file:

theString=unicode(open(excelFile).read(),'UTF-8','replace')

I am then using lxml to do some processing. These files have many tables, the first step of my processing requires that I find the right table. I can find the table based on words that are in the the first cell of the first row. This is where is gets tricky. I had hoped to use a regular expression to test the text_content() of the cell but discovered that there were too many variants of the words (in a test run of 3,200 files I found 91 different ways that the concept that defines just one of the tables was expressed. Therefore I decided to dump all of the text_contents of the particular cell out and use some algorithims in excel to strictly identify all of the variants.

The code I used to write the text_content() was

 headerDict['header_'+str(column+1)]=encode(string,'Latin-1','replace')

I did this baseed on previous answers to questions similar to mine here where it seems the consensus was to read in the file using unicode and then encode it just before the file is written out.

So I processed the labels/words in excel – converted them all to lower case and got rid of the spaces and saved the output as a text file.

The text file has a column of all of the unique ways the table I am looking for is labeled

I then am reading in the file – and the first time I did I read it in using

labels=set([label for label in unicode(open('C:\\balsheetstrings-1.txt').read(),'UTF-8','replace').split('\n')])

I ran my program and discovered that some matches did not occur, investigating it I discovered that unicode replaced certain charactors with \ufffd like in the example below

u'unauditedcondensedstatementsoffinancialcondition(usd\ufffd$)inthousands'

More research turns up that the replacement happens when unicode does not have a mapping for the character (probably not the exact explanation but that was my interpretation)

So then I tried (after thinking what do I have to lose) reading in my list of labels without using unicode. So I read it in using this code:

labels=set(open('C:\\balsheetstrings-1.txt').readlines())

now looking at the same label in the interpreter I see

'unauditedcondensedstatementsoffinancialcondition(usd\xa0$)inthousands'

I then try to use this set of labels to match and I get this error

Warning (from warnings module):
File "C:\FunctionsForExcel.py", line 128
if tableHeader in testSet:
UnicodeWarning: Unicode equal comparison failed to convert both arguments to Unicode - interpreting them as being unequal

Now the frustrating thing is that the value for tableHeader is NOT in the test set When I ask for the value of tableHeader after it broke I received this

'fairvaluemeasurements:'

And to add insult to injury when I type the test into Idle

tableHeader in testSet

it correctly returns false

I understand that the code ‘\xa0’ is code for a non-breaking space. So does Python when I read it in without using unicode. I thought I had gotten rid of all the spaces in excel but to handle these I split them and then joined them

 labels=[''.joiin([word for word in label.split()] for label in labels])

I still have not gotten to a question yet. Sorry I am still trying to get my head around this. It seems to me that I am dealing with inconsistent behavior here. When I read the string in originally and used unicode and UTF-8 all the characters were perserved/transportable if you will. I encoded them to write them out and they displayed fine in Excel, I then saved them as a txt file and they looked okay But something is going on and I can’t seem to figure out where.

If I could avoid writing the strings out to identify the correct labels I have a feeling my problem would go away but there are 20,000 or more labels. I can use a regular expression to cut my potential list down significantly but some of it just requires inspection.

As an aside I will note that the source files all specify the charset=’UTF-8′

Recap- when I read sourcedocument and list of labels in using unicode I fail to make some matches because the labels have some characters replaced by the ufffd, and when I read the sourcedocument in using unicode and the list of labels in without any special handling I get the warning.

I would like to understand what is going on so I can fix it but I have exhausted all the places I can think to look

Answer 1

I understand that the code ‘\xa0’ is code for a non-breaking space.

In a byte string, \xA0 is a byte representing non-breaking space in a few encodings; the most likely of those would be Windows code page 1252 (Western European). But it’s certainly not UTF-8, where byte \xA0 on its own is invalid.

Use .decode('cp1252') to turn that byte string into Unicode instead of 'utf-8'. In general if you want to know what encoding an HTML file is in, look for the charset parameter in the <meta http-equiv="Content-Type"> tag; it is likely to differ depending on what exported it.