Question

So I wrote a small python script that allows me to specify some folder that contains video files, and some output directory, and the program loops over all the video files, and converts them using handbrake like:

proc = subprocess.Popen('HandBrakeCLI -i ... -o ...')
proc.wait()

So it does each file in the directory, one by one. I have a quad core machine, and want to speed things by doing video conversions in parallel but I don’t entirely understand how to approach that.

Do I need something like os.fork()? Something more like the multiprocessing module? I come from single threaded javascript land so this relatively new to me.

Thanks for any input!

Answer 1

Something similar to this should do the trick:

import multiprocessing
...
def wrapper(currentfile):
    #create your commandline -- for the sake of simplicity, I assume that
    #the only thing that changes is the filename you are passing to
    #HandBrakeCLI (and that the filename is the last argument to pass)
    cmd='HandBrakeCLI -i ... -o ... '+currentfile 
    proc = subprocess.Popen(cmd)
    proc.wait()
    return 'foo'

files=os.listdir(path)  #Any way that you build up your list of files is fine
output = multiprocessing.Pool(4).map(wrapper,files) #output is ['foo', 'foo', ..., 'foo']

Of course, this uses a map-like function for it’s side-effects which many python people dislike… but I find it to be intuitive enough — especially if you leave a comment. I also made the function return ‘foo’ to demonstrate that you can access the return value from the function quite easily.