Question

I have a large 100mb file which I would like to perform about 5000 string replacements on it, what is the most efficient way of achieving this?

Is there no better way then reading the file line by line and performing the 5000 replacements on each line?

I also tried reading the file as a string using the .read method when opening the file and performing the 5000 replacements on the string, but this is even slower since it makes 5000 copies of the whole file.

This script has to run on windows using python 2.6

Thanks in advance

Answer 1

Try the following, in this order, until you get one that is fast enough.

1. Read the file into a large string and do each replacement in turn, overwriting the same variable.

   with open(..., 'w') as f:
       s = f.read()
       for src, dest in replacements:
           s = s.replace(src, dest)
       f.seek(0)
       f.write(s)
   

2. Memory map the file, and write a custom replacement function that does the replacements.