Question

I’d like to figure out a way to alert a python script that a file is done copying. Here is the scenario:

1. A folder, to_print is being watched by the script by constantly polling with os.listdir().

2. Every time os.listdir() returns a list of files in which a file exists that hasn’t been seen before, the script performs some operations on that file, which include opening it and manipulating its contents.

This is fine when the file is small, and copying the file from its original source to the directory being watched takes less time than the amount of time remaining until the next poll by os.listdir(). However, if a file is polled and found, but it is still in the process of being copied, then the file contents are corrupt when the script tries to act on it.

Instead, I’d like to be able to (using os.stat or otherwise) know that a file is currently being copied, and wait for it to be done until I act on it if so.

My current idea is to use os.stat() every time I find a new file, then wait until the next poll and compare the date modified/created time since the last time I polled, and if they remain the same then that file is “stable”, otherwise keep polling until it is. I’m not sure this will work though as I am not too familiar with how Linux/Unix updates these values.

Answer 1

Since the files can be copied within the poll interval, just process the new files found by the last poll before checking for new files. In other words, instead of this:

while True:
    newfiles = check_for_new_files()
    process(newfiles)
    time.sleep(pollinterval)

Do this:

newfiles = []

while True:
    process(newfiles)
    newfiles = check_for_new_files()
    time.sleep(pollinterval)

Or just put the wait in the middle of the loop (same effect really):

while True:
    newfiles = check_for_new_files()
    time.sleep(pollinterval)
    process(newfiles)