Question

Lets assume I have a bash script that executes code like so:

for i in $LIST; do
 /path/to/my/script.sh $i &
done

As you can see, I am pushing these scripts into the background, and allowing the parent script to execute as many commands as it can, as fast as it can. The problem is that my system will eventually run out of memory, as these commands take a about 15 or 20 seconds to run each instance.

I’m running one static script.sh file, and passing a simple variable (i.e. customer number) into the script. There are about 20,000 – 40,000 records that I am looping through at any given time.

My question is, how can I tell the system to only have X number of script.sh instances running at once. If too many are running, I want to pause the script until the number of scripts are below the threshold, and then continue.

Any ideas?

Answer 1

A bash-only solution:

MAXJOBS=<your-requested-max + 3>
for i in $LIST; do
 /path/to/my/script.sh $i &
 while true; do
   NUMJOBS=`ps --ppid $$ -o pid= | wc | awk -F ' ' '{ print $1;}'`
   test $NUMJOBS -lt $MAXJOBS && break
done