Question

I have the below code doing everything except what I want :-). The goal is simple, I am trying to add the text to appendtext variable at the top of each file present in the directory (and subdirectories), the script runs fine but the text is not getting appended. Where am I going wrong?

import os
import sys
import fnmatch
temp_fname = "temp_file"

appendtext="""Test string
"""
if len(sys.argv) < 2:
    sys.exit('Usage: test.py <build directory>')

for path,dirs,files in os.walk(sys.argv[1]):
    for fname in files:
        for pat in ['*.*']:
            if fnmatch.fnmatch(fname,pat):
                fullname = os.path.join(path,fname)
                with open(fullname, "r") as in_file:
                    with open(temp_fname, "w") as out_file:
                        out_file.write(appendtext)
                        for line in in_file:
                            out_file.write(line)
                os.rename(temp_fname, fullname)

Answer 1

If you want the text at the top of the file, you should do something like this:

temp_fname = "temp_file"
# the next line doesn't work in Python 2.5, 2.6, or 3.0
with open(fullname, "r") as in_file, open(temp_fname, "w") as out_file:
    out_file.write(appendtext)
    for line in in_file:
        out_file.write(line)
os.rename(temp_fname, fullname)

Here is the above rewritten for Python 2.6:

temp_fname = "temp_file"
with open(temp_fname, "w") as out_file:
    with open(fullname, "r") as in_file:
        out_file.write(appendtext)
        for line in in_file:
            out_file.write(line)
os.rename(temp_fname, fullname)

We can do a little better than this. This always uses the same temporary filename ("temp_file") and that file will always be created in a single directory (the default directory when you run this). What we really want is a temporary file, with a unique name, created in exactly the same directory as the file we will be editing. Python provides us with a handy module called tempfile which creates temporary files.

By default, you just get an open file handle and you don’t know the filename. But we need to know the filename, so that after the temp copy is fully done, we can rename it to the original file name. tempfile provides NamedTemporaryFile for cases like this.

Here is a complete program:

import fnmatch
import os
import sys
import tempfile

headertext = "# header text\n\n"

def want_this_file(fname):
    for pat in ['*']:
        if fnmatch.fnmatch(fname, pat):
            return True
    return False

def prepend_file(fullname, path):
    # with statement means temp file is written and closed at end of with
    with tempfile.NamedTemporaryFile(dir=path, delete=False) as out_file:
        with open(fullname, "r") as in_file:
                out_file.write(headertext)
                for line in in_file:
                    out_file.write(line)
        # before temp file is closed, get its name
        temp_fname = out_file.name
    # rename temp file to fullname, clobbering original
    os.rename(temp_fname, fullname)


start_directory = sys.argv[1]

for dirpath, dirnames, filenames in os.walk(start_directory):
    for fname in filenames:
        if want_this_file(fname):
            fullname = os.path.join(dirpath, fname)
            prepend_file(fullname, dirpath)

This answer uses the pattern “write a temp file, then rename the temp file to the original name”. This is the way you should do it. It lets the code write the new version, and only when the new version is fully and successfully written, then does a single action to rename the new file to the old filename. Thus, if anything goes wrong while trying to write the new version, the original file is left untouched. This is a safe way to solve the problem.

We want to create the temp file in the same directory as the original file, so that the os.rename() operation will be trivially cheap. On a Linux system, your system temp directory (/tmp) might be on its own partition, and if you just let tempfile create its temporary file there, then the rename operation might involve copying the data again! If the temp file is in the same directory, the rename operation is always very fast and safe.

EDIT: Here is an improved version of the code. This catches errors, and cleans up the temp file before re-raising the exception signalling the error. Also, as J.F. Sebastian pointed out, the files should be opened in binary mode; this does that.

import fnmatch
import os
import shutil
import sys
import tempfile

file_patterns_to_match = ['*']

headertext = "# header text\n\n"
# make any newlines in headertext match the system line ending
headertext = headertext.replace('\n', os.linesep)

def want_this_file(fname):
    for pat in file_patterns_to_match:
        if fnmatch.fnmatch(fname, pat):
            return True
    return False

def prepend_file(fullname, path):
    # with statement means temp file is written and closed at end of with
    with tempfile.NamedTemporaryFile(dir=path, delete=False) as out_file:
        # get the name immediately
        temp_fname = out_file.name

        try:
            # use binary mode to avoid newline translations
            with open(fullname, "rb") as in_file:
                out_file.write(headertext)
                shutil.copyfileobj(in_file, out_file)
        except Exception:
            # on any error, clean up temp file and re-raise exception
            try:
                os.remove(temp_fname)
            except Exception:
                print("unable to clean up temp file: " + temp_fname)
                pass
            raise
    # rename temp file to fullname, clobbering original
    os.rename(temp_fname, fullname)


start_directory = sys.argv[1]

for dirpath, dirnames, filenames in os.walk(start_directory):
    for fname in filenames:
        if want_this_file(fname):
            fullname = os.path.join(dirpath, fname)
            prepend_file(fullname, dirpath)