Question

Consider the following script:

for host in $(get-all-hosts)
do
  (restart-server $host; wait-for-server-to-come-up $host) &
done

As you might guess, restart-server restarts the server and the command wait-for-server-to-come-up blocks until the server is up (e.g., grep -m 1 'server up' <(tail -f /path/to/log)).

This script essentially restarts all servers simultaneously. I’m curious what’s the simplest way to modify this script to stop after some fixed number of servers, and wait for one server to come up before proceeding with the next restart so that a maximum of, say, 4 servers is down at any given time. One way I know of doing this is to simply restart in chunks of 4 and wait on all of the pids in each chunk, but I’m hoping it’s not that hard to do something smarter.

Some scratch work toward a solution:

Second try, using some ideas from Dennis’s link. Almost ideal for a vanilla Bash solution:

mkfifo mfifo
exec 3<>mfifo
echo >&3
echo >&3
echo >&3
for host in $(get-all-hosts)
do
  read
  (restart-server $host; wait-for-server-to-come-up $host; echo >&3) &
done <&3

My biggest remaining complaint with this solution is that it assumes there isn’t already a named queue called mfifo that’s already in use. Other than that, I haven’t seen any problems, and it works exactly as expected as far as I can tell.

Answer 1

xargs has a parallelism feature similar to parallel:

echo $(get-all-hosts) | tr ' ' '\n' | xargs -P 4 -n 1 ./blocking-restart

..where blocking-restart is expected to take the name of a single server, restart it and wait until it’s finished. Note the tr puts each host on its own line, which is what xargs expects.