Question

I have a fortran atoms.out file that performs some function. I am calling that fortran file from within python with:

values = subprocess.Popen(["./atoms.out", "./%s.klj" % test1, "./%s.out" % test1], stdout = subprocess.PIPE)

I’ve looked at Python PP, but I dont think it is what I am looking for, as it uses n number of CPUs for a single job.

The calculations conducted by the fortran file are not independent and cannot be used with python pp

So my question is how can i submit a number of jobs to a number of different CPUs, say for example I have: 7 Files that I want to be run over 4 CPUs.

So each job is distributed as:

File1 -> CPU1
File2 -> CPU2
File3 -> CPU3
File4 -> CPU4

Then when a job has finished on a CPU it is replaces with the remaining 3 Files to run. Say File1 is finished on CPU1 it is replaced by File5, so the distribution of jobs would look like:

File5 -> CPU1
File2 -> CPU2
File3 -> CPU3
File4 -> CPU4

And so forth until it has finished running the fortran file on the 7 files.

Answer 1

Use the multiprocessing.pool.ThreadPool class to create a thread pool consisting of (in your case) four threads. Submit the subprocess.Popen task to the pool using e.g. pool.map. Then, the four threads in Python will spawn one Fortran process each.