Question

I have several threads running in parallel from Python on a cluster system. Each python thread outputs to a directory mydir. Each script, before outputting checks if mydir exists and if not creates it:

if not os.path.isdir(mydir):
    os.makedirs(mydir)

but this yields the error:

os.makedirs(self.log_dir)                                             
  File "/usr/lib/python2.6/os.py", line 157, in makedirs
mkdir(name,mode)
OSError: [Errno 17] File exists

I suspect it might be due to a race condition, where one job creates the dir before the other gets to it. Is this possible? If so, how can this error be avoided?

I’m not sure it’s a race condition so was wondering if other issues in Python can cause this odd error.

Answer 1

Any time code can execute between when you check something and when you act on it, you will have a race condition. One way to avoid this (and the usual way in Python) is to just try and then handle the exception

while True:
    mydir = next_dir_name()
    try:
        os.makedirs(mydir)
        break
    except OSError, e:
        if e.errno != errno.EEXIST:
            raise   
        # time.sleep might help here
        pass

If you have a lot of threads trying to make a predictable series of directories this will still raise a lot of exceptions, but you will get there in the end. Better to just have one thread creating the dirs in that case