Question

We have 10 Linux boxes that must run 100 different tasks each week. These computers work at these tasks mostly at night when we are at home. One of my coworkers is working on a project to optimize run time by automating the starting of the tasks with Python. His program is going to read a list of tasks, grab an open task, mark that task as in progress in the file, and then once the task is finished mark the task as complete in the file. The task files will be on our network mount.

We realize it is not recommended to have multiple instances of a program accessing the same file, but we don’t really see any other option. While he was looking for a way of preventing 2 computers from writing to the file at the same time, I came up with a method of my own, which seemed easier to implement than the methods we found online. My method is to check if the file exists, wait a few seconds if it doesn’t, and then temporarily move the file if it does. I wrote a script to test this method:

#!/usr/bin/env python

import time, os, shutil
from shutil import move
from os import path


fh = "testfile"
fhtemp = "testfiletemp"


while os.path.exists(fh) == False:
    time.sleep(3)

move(fh, fhtemp)
f = open(fhtemp, 'w')
line = raw_input("type something: ")
print "writing to file"
f.write(line)
raw_input("hit enter to close file.")
f.close()
move(fhtemp, fh)

In our tests this method has worked, but I was wondering if we might have some problems that we aren’t seeing by using this. I realize that disaster could result from two computers running exists() at the same time. It’s unlikely that two computers would get to that point at the same time since the tasks are anywhere between 20 minutes and 8 hours.

Answer 1

You’ve basically developed a filesystem version of the binary semaphore (or mutex). It’s a well-studied structure used for locking, so as long as you get the implementation details right, it should work. The trick is to get the “test and set” operation, or in your case “check existence and move,” to be truly atomic. For that I’d use something like this:

lock_acquired = False
while not lock_acquired:
    try:
        move(fh, fhtemp)
    except:
        sleep(3)
    else:
        lock_acquired = True
# do your writing
move(fhtemp, fh)
lock_acquired = False

The program as you had it would work most of the time, but as mentioned you could have issues if another process moved the file between the check for its existence and the call to move. I suppose you could work around that, but I’d personally recommend sticking with a well-tested mutex algorithm. (I’ve translated/ported the above code sample from Modern Operating Systems by Andrew Tanenbaum, but it’s possible that I’ve introduced errors in the conversion – just fair warning)

By the way, the man page for the open function on Linux offers this solution for file locking:

The solution for performing atomic file locking using a lockfile is to create a unique file on the same file system (e.g., incorporating hostname and pid), use link(2) to make a link to the lockfile. If link() returns 0, the lock is successful. Otherwise, use stat(2) on the unique file to check if its link count has increased to 2, in which case the lock is also successful.

To implement that in Python, you could do something like this:

# each instance of the process should have a different filename here
process_lockfile = '/path/to/hostname.pid.lock'
# all processes should have the same filename here
global_lockfile = '/path/to/lockfile'
# create the file if necessary (only once, at the beginning of each process)
with open(process_lockfile, 'w') as f:
    f.write('\n') # or maybe write the hostname and pid

# now, each time you have to lock the file:
lock_acquired = False
while not lock_acquired:
    try:
        link(process_lockfile, global_lockfile)
    except:
        lock_acquired = (stat(process_lockfile).st_nlinks == 2)
    else:
        lock_acquired = True
# do your writing
unlink(global_lockfile)
lock_acquired = False