Question

I’ve got a script that downloads and then converts videos. I’m trying to do all of the downloads at once(a lot of wgets) and then when they’re done, convert them. Right now I have to wait while each file downloads individually and then convert when done. I want all the download requests to run concurrently.

Here’s the part in my ‘download’ script that is holding everything up.

for id  in ids:
    subprocess.Popen(cmd, shell=True).wait()

I have my ‘convert’ script waiting on the download script to finish. (which is necessary, because i’m using sets of downloads to organize everything.)

I could use a queue but this I’ve already made this a big enough mess and I’m hoping there is a simpler solution.

Thanks

Answer 1

you could use

process_list = []    
for id  in ids:
    process_list.append(subprocess.Popen(cmd, shell=True))
for process in process_list:
    process.wait()

as such you will wait just at the end of simultaneous jobs.