Question

Related to a previous question, I’m trying to do replacements over a number of large CSV files.

The column order (and contents) change between files, but for each file there are about 10 columns that I want and can identify by the column header names. I also have 1-2 dictionaries for each column I want. So for the columns I want, I want to use only the correct dictionaries and want to implement them sequentially.

An example of how I’ve tried to solve this:

# -*- coding: utf-8 -*-
import re

# imaginary csv file. pretend that we do not know the column order.
Header = [u'col1', u'col2']
Line1 = [u'A',u'X']
Line2 = [u'B',u'Y']
fileLines = [Line1,Line2]

# dicts to translate lines
D1a = {u'A':u'a'}
D1b = {u'B':u'b'}
D2 = {u'X':u'x',u'Y':u'y'}

# dict to correspond header names with the correct dictionary.
# i would like the dictionaries to be read sequentially in col1.
refD = {u'col1':[D1a,D1b],u'col2':[D2]}

# clunky replace function
def freplace(str, dict):
    rc = re.compile('|'.join(re.escape(k) for k in dict))
    def trans(m):
        return dict[m.group(0)]
    return rc.sub(trans, str)

# get correspondence between dictionary and column
C = []
for i in range(len(Header)):
    if Header[i] in refD:
        C.append([refD[Header[i]],i])

# loop through lines and make replacements
for line in fileLines:
    for i in range(len(line)):
        for j in range(len(C)):
            if C[j][1] == i:
                for dict in C[j][0]:
                    line[i] = freplace(line[i], dict)

My problem is that this code is quite slow, and I can’t figure out how to speed it up. I’m a beginner, and my guess was that my freplace function is largely what is slowing things down, because it has to compile for each column in each row. I would like to take the line rc = re.compile('|'.join(re.escape(k) for k in dict)) out of that function, but don’t know how to do that and still preserve what the rest of my code is doing.

Answer 1

You don’t need re:

# -*- coding: utf-8 -*-

# imaginary csv file. pretend that we do not know the column order.
Header = [u'col1', u'col2']
Line1 = [u'A',u'X']
Line2 = [u'B',u'Y']
fileLines = [Line1,Line2]

# dicts to translate lines
D1a = {u'A':u'a'}
D1b = {u'B':u'b'}
D2 = {u'X':u'x',u'Y':u'y'}

# dict to correspond header names with the correct dictionary
refD = {u'col1':[D1a,D1b],u'col2':[D2]}

# now let's have some fun...

for line in fileLines:
    for i, (param, word) in enumerate(zip(Header, line)):
        for minitranslator in refD[param]:
            if word in minitranslator:
                line[i] = minitranslator[word]

returns:

[[u'a', u'x'], [u'b', u'y']]