Question

Possible Duplicate:
Fastest Way to Delete a Line from Large File in Python
How to edit a line in middle of txt file without overwriting everything?

I know I can read every line into a list, remove a line, then write the list back.

But the file is large, is there a way to remove a part in the middle of the file, and needn’t rewrite the whole file?

Answer 1

I don’t know if a way to change the file in place, even using low-level file system commands, but you don’t need to load it into a list, so you can do this without a large memory footprint:

with open('input_file', 'r') as input_file:
    with open('output_file', 'w') as output_file:
        for line in input_file:
            if should_delete(line):
                pass
            else:
                output_file.write(line)

This assumes that the section you want to delete is a line in a text file, and that should_delete is a function which determines whether the line should be kept or deleted. It is easy to change this slightly to work with a binary file instead, or to use a counter instead of a function.

Edit: If you’re dealing with a binary file, you know the exact position you want to remove, and its not too near the start of the file, you may be able to optimise it slightly with io.IOBase.truncate (see http://docs.python.org/2/library/io.html#io.IOBase). However, I would only suggest pursuing this if the a profiler indicates that you really need to optimise to this extent.