Question

I have a file that has over 200 lines in this format:

name old_id new_id

The name is useless for what I’m trying to do currently, but I still want it there because it may become useful for debugging later.

Now I need to go through every file in a folder and find all the instances of old_id and replace them with new_id. The files I’m scanning are code files that could be thousands of lines long. I need to scan every file with each of the 200+ ids that I have, because some may be used in more than one file, and multiple times per file.

What is the best way to go about doing this? So far I’ve been creating python scripts to figure out the list of old ids and new ids and which ones match up with each other, but I’ve been doing it very inefficient because I basically scanned the first file line by line and got the current id of the current line, then I would scan the second file line by line until I found a match. Then I did this over again for each line in the first file, which ended up with my reading the second file a lot. I didn’t mind doing this inefficiently because they were small files.

Now that I’m searching probably somewhere around 30-50 files that can have thousands of line of code in it, I want it to be a little more efficient. This is just a hobbyist project, so it doesn’t need to be super good, I just don’t want it to take more than 5 minutes to find and replace everything, then look at the result and see that I made a little mistake and need to do it all over again. Taking a few minutes is fine(although I’m sure with computers nowadays they can do this almost instantly still) but I just don’t want it to be ridiculous.

So what’s the best way to go about doing this? So far I’ve been using python but it doesn’t need to be a python script. I don’t care about elegance in the code or way I do it or anything, I just want an easy way to replace all of my old ids with my new ids using whatever tool is easiest to use or implement.

Examples:

Here is a line from the list of ids. The first part is the name and can be ignored, the second part is the old id, and the third part is the new id that needs to replace the old id.

unlock_music_play_grid_thumb_01 0x108043c 0x10804f0

Here is an example line in one of the files to be replaced:

const v1, 0x108043c

I need to be able to replace that id with the new id so it looks like this:

const v1, 0x10804f0

Answer 1

Use something like multiwordReplace (I’ve edited it for your situation) with mmap.

import os
import os.path
import re
from mmap import mmap
from contextlib import closing


id_filename = 'path/to/id/file'
directory_name = 'directory/to/replace/in'

# read the ids into a dictionary mapping old to new
with open(id_filename) as id_file:
    ids = dict(line.split()[1:] for line in id_file)    

# compile a regex to do the replacement
id_regex = re.compile('|'.join(map(re.escape, ids)))

def translate(match):
    return ids[match.group(0)]

def multiwordReplace(text):
    return id_regex.sub(translate, text)

for code_filename in os.listdir(directory_name):
    with open(os.path.join(directory, code_filename), 'r+') as code_file:
        with closing(mmap(code_file.fileno(), 0)) as code_map:
            new_file = multiword_replace(code_map)
    with open(os.path.join(directory, code_filename), 'w') as code_file:
        code_file.write(new_file)