Question

I have a problem where I run scripts of my bash command line one after the other like:

./process.sh data1.txt
./process.sh data2.txt
...
./process.sh dataN.txt

What I’d like to do is start these as background jobs but only have 10 running at any given time. I can do this in a traditional programming language (am very comfortable using joins, wait style constructs) the question I have is how do I do this using UNIX bash.

Thanks.

Answer 1

BG_LIMIT=10
counter=0
for data in data*.txt; do
    ./process.sh $data &
    if (( ++counter % $BG_LIMIT == 0 )); then
        echo "Wating for background jobs to complete..."
        wait
    fi
done
echo "Waiting for any remaining jobs to complete..."
wait