Question

I’m currently working on a Python script that takes a list of log files (from a search engine) and produces a file with all the queries within these, for later analysis.
Another feature of the script is that it removes the most common words, which I’ve also implemented, but I’ve faced a problem I can’t seem to overcome. The removing of words does work as intended, as long as the queries does not contain special characters. As the search logs are in Danish, the characters æ, ø and å will appear regularly.

Searching on the topic I’m now aware that I need to encode these into UTF-8, which I’m doing when obtaining the query:

tmp = t_query.encode("UTF-8").lower().split()

t_query is the query and I split it up to later compare each word with my list of forbidden words. If I do not use the encoding I’ll get the error:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe6 in position 1: ordinal not in range(128)

Edit: I also tried using the decode instead, but get the following error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xa7' in position 3: ordinal not in range(128)

I loop through the words like this:

for i in tmp:
    if i in words_to_filter:
        tmp.remove(i)

As said this works perfectly for words not including special characters. I’ve tried to print the i along with the current forbidden word and will get e.g:

færdelsloven - færdelsloven

Where the first word is the ith element in tmp. The last word in the one from the forbidden words. Obviously something has gone wrong, but I just can’t manage to find a solution. I’ve tried many suggestions found on Google and in here, but nothing have worked so far.

Edit 2: if it makes a difference, I’ve tried loading the log files both with and without the use of codec:

with codecs.open(file_name, "r", "utf-8") as f_src:
    jlogs = map(json.loads, f_src.readlines())

I’m running Python 2.7.2 from a Windows environment, if it matters. The script should be able to run on other platforms (namely Linux and Mac OS).

I would really appreciate if one of you are able to help me out.

Best regards
Casper

Answer 1

I’ve finally solved it. As Lattyware Python 3.x seems to do much better. After changing the version and encoding the Python file to Unicode it works as intended.