Question

After hours of fighting with a script, I narrowed the problem down to the following. Running this at the command prompt (bash) to shows the issue:

bash$ echo "\n"
\n
bash$ echo "\n" | sed 's/\\n/---\\&/g'
---\\n
bash$ str=`echo "\n" | sed 's/\\n/---\\&/g'`
bash$ echo $str
\n
bash$ 

I don’t understand why that str=... command is not equivalent to str= output of command in ....

Why is it not equivalent to str=”—\n” ?

I must be missing something fundamental here.

EDIT: In response to comments:

$ /bin/bash --version
GNU bash, version 4.2.8(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2011 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

also, the solution isn’t as simple as the answer stating only the first part of the pipe is returned in a backtick.
Notice:

bash$ str=`echo "abc" | sed 's/a/---&/g'`
bash$ echo $str
---abc
bash$

So somehow it is a combination of things including \ characters that are causing the problem. I can’t quite figure out the behavior here.

Answer 1

Other answer is probably correct about some quoting being dropped. The trick that I’ve learned lately is to use $() instead of backticks.

str=$(echo "\n" | sed 's/\\n/---\\&/g')

That works as expected.

---

Update

Here is how it seems to work. Consider this example first:

$ echo "\\n"
\n
$ echo '\\n'
\\n

In shell there is strong and weak quoting. Weak quoting is represented via ” (double quote) and strong quoting is represented via ‘ (single quote). Weak quoting does not prevent escaping while strong will prevent it. And back slash has the meaning of preserving next character except newline. So “\\n” is actually equal to two characters: \ and n because first slash is escaped. Which are printed. Strong quotes, on the other hand, will not do any escaping and ‘\\n’ becomes three characters. \, \ and n.

One other property of weak quotes is that they prevent strong quotes from taking effect and make them be interpreted literally.

So now, consider example similar to your sed example:

$ echo `echo '\\n'`
\n

So what happens here? back ticks are processed as weak quotes, that is strong quotes inside stop doing any quoting. So ‘\\n’ gets interpreted as ‘\n’ since \ gets escaped. As a result, command that gets executed is echo '\n' and not echo '\\\\n'. And that’s exactly what happened with you sed.

Awesome question, really made me think and research it.