Question

I’m trying to get this function for making it easy to parallelize my bash scripts working. The idea is simple; instead of running each command sequentially, I pipe the command I want to run to this function and it does while read line; run the jobs in the bg for me and take care of logistics…. it doesn’t work though. I added set -x by where stuff’s executed and it looks like I’m getting weird quotes around the stuff I want executed… what should I do?

runParallel () {
    while read line
    do
        while [ "`jobs | wc -l`" -eq 8 ]
        do
            sleep 2
        done

        {
            set -x
            ${line}
            set +x
        } &
    done

    while [ "`jobs | wc -l`" -gt 0 ] 
    do
        sleep 1
        jobs >/dev/null 2>/dev/null
        echo sleeping
    done
}

for H in `ypcat hosts | grep fmez | grep -v mgmt | cut -d\  -f2 | sort -u`
do
    echo 'ping -q -c3 $H 2>/dev/null 1>/dev/null && echo $H - UP || echo $H - DOWN'
done | runParallel

When I run it, I get output like the following:

> ./myscript.sh
+ ping -q -c3 '$H' '2>/dev/null' '1>/dev/null' '&&' echo '$H' - UP '||' echo '$H' - DOWN
Usage: ping [-LRUbdfnqrvVaA] [-c count] [-i interval] [-w deadline]
            [-p pattern] [-s packetsize] [-t ttl] [-I interface or address]
            [-M mtu discovery hint] [-S sndbuf]
            [ -T timestamp option ] [ -Q tos ] [hop1 ...] destination
+ set +x
sleeping
> 

Answer 1

The quotes in the set -x output are not the problem, at most they are another result of the problem. The main problem is that ${line} is not the same as eval ${line}.

When a variable is expanded, the resulting words are not treated as shell reserved constructs. And this is expected, it means that eg.

A="some text containing > ; && and other weird stuff"
echo $A

does not shout about invalid syntax but prints the variable value.

But in your function it means that all the words in ${line}, including 2>/dev/null and the like, are passed as arguments to ping, which set -x output nicely shows, and so ping complains.

If you want to execute from variables complicated commandlines with redirections and conditionals, you will have to use eval.