Question

I’m constantly confused by the rules for quoting and evaluating when I’m writing bash scripts. I know some of the basics, like the difference between ” and “” and “, but I still seem to get it wrong far too often, and be reduced to experimenting with trying all sorts of different ways to say the same thing.

Any individual problem I can usually work out by brute force, but I think my conceptual model of how it works must be hopelessly broken in some unknown way.

I have no problem with lisp’s quote,eval,read,print,syntax-quote system. In fact I wrote a little kata to help people understand what was going on: http://www.learningclojure.com/2010/11/syntax-quote-kata-for-confused.html

I guess that I’m looking for something similar for bash, (which seems much more complicated). A good model, or a set of exercises that will help me form such a model, that will make me able to look at a complicated shell script where variables are being transformed and evaluated and printed out and read, and work out what’s going to happen without having to try it.

Failing that, good methods of debugging the process, and watching what’s going on at each stage of the evaluation would be really helpful.

Answer 1

Bruce Barnett’s UNIX Shell Quote Tutorial is awesome, and the Bash FAQ/pitfalls/word splitting articles have tons of useful tips. A short summary:

Unquoted strings can contain most characters, but not all (like newlines) , and many of them (including space) will have to be escaped. Just don’t use them – If you fall for the temptation you might find that someone who modified the script forgot to include quotes once they became necessary.

Single quoted strings can contain most characters, including NUL and newlines, but not single quotes, so they are also useful only for simple values.

Backticks are for commands. They should only be used if your shell does not support $(). Example:

current_dir=`pwd` # BAD! Don't do this!

That command is bad, because when the right hand side of an assignment is not quoted the shell performs word splitting on it. It often leads to hard-to-reproduce bugs, because whitespace is difficult to check visually. To quote commands you have to use double quotes:

current_dir="$(pwd)" # OK, but loses newlines at EOF

Newlines at EOF are especially tricky. You can add a single character and strip it by using for example

# Works for some commands, but not pwd
current_dirx="$(pwd; echo x)"
current_dir="${current_dirx%x}"
printf %s "$current_dir"

, but there’s an additional difficulty because some commands (like pwd) will add a newline at the end of their output anyway, so you might have to remove that as well:

# Works for some commands, including pwd
current_dirx="$(pwd; echo x)"
current_dir="${current_dirx%$'\nx'}"
printf %s "$current_dir"

Double quotes can contain any character (Try echo -ne "\0" | wc -c), but note that variables can’t contain the NUL character.

ANSI-C quotes can contain any characters except NUL (Try echo -ne $'\0' | wc -c), and provides handy escape codes to make it easier to work with special characters:

printf %s $'--$`!*@\a\b\E\f\r\t\v\\\'"\360\240\202\211 \n'
printf %q $'--$`!*@\a\b\E\f\r\t\v\\\'"\360\240\202\211 \n'
touch -- $'--$`!*@\a\b\E\f\r\t\v\\\'"\360\240\202\211 \n'
rm -- $'--$`!*@\a\b\E\f\r\t\v\\\'"\360\240\202\211 \n'