Question

I want to read and write python-source-files from the file system in a thread-safe way.

open("n2.py","w").write(my_new_py_class)
from myproject import n2
#do something with n2

I assume that this is not thread-safe, since a request2 could modify the file before request1 is loading and executing it. I would like to achieve something like that one thread is waiting till the other thread wrote, loaded, executed and closed the file.

Answer 1

Why are you making your application modify its own files? This is not only incredibly evil, metaprogramming is orders of magnitude harder to understand debug. Plus, python caches modules it imports, so it’s not really easy to reload that. And, last but not least, you don’t have to writ the code to a file to execute it, if you really must execute dynamically generated code.

To answer your question about writing files in a thread safe way, the general convention is to:

• Write your content to a temporary file on the same filesystem as your target file.
• Rename that temporary file to your target file, overwritting it in the process.

This works, because rename is atomic on POSIX systems, when done on the same device. So other threads/processes will either still have the old file opened, which is now detached from the filesystem and will be deleted as soon as those threads/processes are done with it, or they will open the new file with all of its contents. You avoid having a file that is only half-written.

In practice, I like to make a temporary directory with python’s tempfile module, and write the file in there, then move it and remove the directory — this way the file is being created with default umask.

Last but not least, rename is not really atomic on Windows, at least with default settings, as it won’t let you overwrite your old file — you need to do two renames, and that introduces a possibility of race condition. I don’t know a good solution for Windows.