Question

I have a shell var called $JOB_COMMAND that has the command to launch a map reduce job like so:

user@server:/scriptpath# echo $JOB_COMMAND
/var/elastic-mapreduce/elastic-mapreduce --create 
    --name 'Extrabux-MapReduce' 
    --num-instances 16 --instance-type c1.medium --key-pair extrabux-keypair 
    --log-uri s3n://extrabux-log/
    --bootstrap-action "s3://extrabux-scripts/startup.sh" |
    egrep -o 'j-[a-zA-Z0-9]+'

Obviously lots of those options are not needed for the point of the question, but in some off chance the content affects the answer, it’s included.

Now when I try to run the command from the variable, I get this:

user@server:/scriptpath# $JOB_COMMAND
Error: --output must follow one of --streaming

However if I copy and paste the result of echoing the command like so I get the following:

user@server:/scriptpath#  /var/elastic-mapreduce/elastic-mapreduce --create 
    --name 'Extrabux-MapReduce' 
    --num-instances 16 --instance-type c1.medium --key-pair extrabux-keypair 
    --log-uri s3n://extrabux-log/
    --bootstrap-action "s3://extrabux-scripts/startup.sh" |
    egrep -o 'j-[a-zA-Z0-9]+'

j-3FV0MT3H7G21S

As you can see, the command runs and successfully spits out a job id

Here is what I would like to do, get the job ID into another shell var:

user@server:/scriptpath# JOBID=`$JOB_COMMAND`

Any idea why running the command when it is inside a var is breaking?
FYI, I have also tried

user@server:/scriptpath# JOBID=$($JOB_COMMAND)

and that also gives me the error

Thanks for your help!

Answer 1

The problem is that the parsing of the pipeline (the | in your example) happens before the variable $JOB_COMMAND is expanded. In Bash, parsing the line happens first, then variables are expanded, split based on $IFS, and substituted into the commands. So in your command, the | egrep and everything following it are being passed as literal arguments to your elastic-mapreduce command; the error about --output is probably due to the -o argument.

As the Bash FAQ points out, you shouldn’t be storing commands in variables, you should use shell functions instead.

job_command () {
    /var/elastic-mapreduce/elastic-mapreduce --create \
        --name 'Extrabux-MapReduce' \
        --num-instances 16 --instance-type c1.medium --key-pair extrabux-keypair \
        --log-uri s3n://extrabux-log/ \
        --bootstrap-action "s3://extrabux-scripts/startup.sh" | \
        egrep -o 'j-[a-zA-Z0-9]+'
}

And now you can call it as follows:

JOBID=$(job_command)

Using shell functions would also allow you to pass parameters in to your command, which you may find to be useful.

If you really want to store a command in a variable for some reason, you will need to use eval, but this is not recommended:

eval $JOB_COMMAND