Question

I have a script that runs some data processing command 10K times.

foreach f (folderName/input*.txt)
  mycmd $f
end

I have timed the runtime for each “mycmd $f” to be 0.25 secs.
With 10K runs, it adds up to be more than 1 hr.
I’m running it on a 16 cores nehalem.
It’s a huge waste to not run on the remaining 15 cores.

I have tried & with sleep, somehow the script just dies with a warning or error around 3900 iterations, see below. The shorter the sleep, that faster it dies.

foreach f (folderName/input*.txt)
  mycmd $f & ; sleep 0.1
end

There has got to be a better way.
Note: I would prefer shell script solutions, let’s not wander into C/C++ land.

Thanks

Regards

Answer 1

Pipe the list of files to

xargs -n 1 -P 16 mycmd

For example:

echo folderName/input*.txt | xargs -n 1 -P 16 mycmd