Question

I have a multithreaded python application that does a bunch of processing on files.

The main application is scanning a directory for new files. Whenever a file is found, this file is being processed (Image/Video editing, FTP uploading, saving metadata to a DB…) in other Python threads. A large part of those Processors are running external commands with Popen and waiting for them to return.

When I process a single file, I don’t have any problem and everything works fine, but when the main application is running, it leads to a random behavior and I end up having a bunch of errors like Popen command not returning, FTP server dropping the connection, etc…

Here is a simplified version of my application, for illustration purposes:

class VideoEditor(Thread):
    def __init__(self, file, manager):
        Thread.__init__(self);
        self.manager = manager
        self.file = file


    def run(self):
        Popen('some video editing command').wait()

        self.manager.on_video_editing_finished(self)

class FileUploader(Thread):
    def __init__(self, file, manager):
        Thread.__init__(self);
        self.manager = manager
        self.file = file

    def run(self):
        # Uploading the file to ftp server

        self.manager.on_upload_finished(self)

class Manager():
    def __init__(self):
        self.video_editors = []
        self.uploaders = []

    def on_upload_finished(self, uploader):
        self.uploaders.remove(uploader)
        file = uploader.file
        print "Processing finished for %s" % file


    def on_video_editing_finished(self, video_editor):
        self.video_editors.remove(video_editor)
        file = video_editor.file
        u = FileUploader(file, self)
        u.start()
        self.uploaders.append(u)

    def scan_and_process(self):
        # Detect new file
        ve = VideoEditor(new_file, self)
        ve.start()
        self.video_editors.append(ve)

if __name__ == "__main__":
    manager = Manager()
    while True:
        manager.scan_and_process()
        sleep(60)

Is there a better design for doing this? Am I doing something wrong here?

EDIT: Idea
Would this way be better?

def process_file(file):
    Popen('some video editing command').wait()
    Popen('some other command').wait()
    # Upload to FTP here

class Manager():
    def scan_and_process(self):
        # Detect new file
        Popen(['python', 'Manager.py', '-process_file', new_file])

if __name__ == "__main__":
    if len(argv) == 3 and argv[1] == 'process_file':
        process_file(argv[2])
    else:
        manager = Manager()
        while True:
            manager.scan_and_process()
            sleep(60)

Thanks!

Answer 1

You don’t need threads for this becuase you’re already using subprocesses. Just let the subprocesses run and use Popen.poll() to check for completion.