Question

Why does GNU sed sometimes handle substitution with piped output into another sed instance differently than when multiple expressions are used with the same one?

Specifically, for msys/mingw sessions, in the /etc/profile script I have a series of manipulations that “rearrange” the order of the environment variable PATH and removes duplicate entries.

Take note that while normally sed treats each line of input seperately (and therfore can’t easily substitute ‘\n’ in the input stream, this sed statement does a substitution of ‘:’ with ‘\n’, so it still handles the entire input stream like one line (with ‘\n’ characters in it). This behavior stays true for all sed expressions in the same instance of sed (basically until you redirect or pipe the output into another program).

Here’s the obligatory specs:

    Windows 7 Professional Service Pack 1
    HP Pavilion dv7-6b78us
    16 GB DDR3 RAM
    MinGW-w64 (x86_64-w64-mingw32-gcc-4.7.1.2-release-win64-rubenvb) mounted on /mingw/
    MSYS (20111123) mounted on / and on /usr/
    $ uname -a="MINGW32_NT-6.1 CHRIV-L09 1.0.17(0.48/3/2) 2011-04-24 23:39 i686 Msys"
    $ which sed="/bin/sed.exe" (it's part of MSYS)
    $ sed --version="GNU sed version 4.2.1"

This is the contents of PATH before manipulation:

    PATH='.:/usr/local/bin:/mingw/bin:/bin:/c/PHP:/c/Program Files (x86)/HP SimplePass 2011/x64:/c/Program Files (x86)/HP SimplePass 2011:/c/Windows/system32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0:/c/si:/c/android-sdk:/c/android-sdk/tools:/c/android-sdk/platform-tools:/c/Program Files (x86)/WinMerge:/c/ntp/bin:/c/GnuWin32/bin:/c/Program Files/MySQL/MySQL Server5.5/bin:/c/Program Files (x86)/WinSCP:/c/Program Files (x86)/Overlook Fing 2.1/bin:/c/Program Files/7-zip:.:/c/Program Files/TortoiseGit/bin:/c/Program Files (x86)/Git/bin:/c/VS10/VC/bin/x86_amd64:/c/VS10/VC/bin/amd64:/c/VS10/VC/bin'

This is an excerpt of /etc/profile (where I have begun the PATH manipulation):

    set | grep --color=never ^PATH= | sed -e "s#^PATH=##" -e "s#'##g" \
    -e "s/:/\n/g" -e "s#\n\(/[^\n]*tortoisegit[^\n]*\)#\nZ95-\1#ig" \
    -e "s#\n\(/[a-z]/win\)#\nZ90-\1#ig" -e "s#\n\(/[a-z]/p\)#\nZ70-\1#ig" \
    -e "s#\.\n#A10-.\n#g" -e "s#\n\(/usr/local/bin\)#\nA15-\1#ig" \
    -e "s#\n\(/bin\)#\nA20-\1#ig" -e "s#\n\(/mingw/bin\)#\nA25-\1#ig" \
    -e "s#\n\(/[a-z]/vs10/vc/bin\)#\nA40-\1#ig"

The last sed expression in that line basically looks for lines that begins with “/c/VS10/VC/bin” and prepends them with ‘A40-‘ like this:

    ...
    /c/si
    A40-/c/VS10/VC/bin
    A40-/c/VS10/VC/bin/amd64
    A40-/c/VS10/VC/bin/x86_amd64
    /c/GnuWin32/bin
    ...

I like my sed expressions to be flexible (path structures change), but I don’t want it to match the lines that end with amd64 or x86_amd64 (those are going to have a different string prepended). So I change the last expression to:

    -e "s#\n\(/[a-z]/vs10/vc/bin\)\n#\nA40-\1\n#ig"

This works:

    ...
    /c/si
    A40-/c/VS10/VC/bin
    /c/VS10/VC/bin/amd64
    /c/VS10/VC/bin/x86_amd64
    /c/GnuWin32/bin
    ...

Then, (to match any “line” matching the pseudocode “/x/…/bin”) I change the last expression to:

    -e "s#\n\(/[a-z]/.*/bin\)\n#\nA40-\1\n#ig"

Which produces:

    ...
    /c/si
    /c/VS10/VC/bin
    /c/VS10/VC/bin/amd64
    /c/VS10/VC/bin/x86_amd64
    /c/GnuWin32/bin
    ...

??? – sed didn’t match any character (‘.’) any number of times (‘*’) in the middle of the line ???

But, if I pipe the output into a different instance of sed (and compensate for sed handling each “line” seperately) like this:

    | sed -e "s#^\(/[a-z]/.*/bin\)$#A40-\1#ig"

I get:

    sed: -e expression #1, char 30: unterminated `s' command

??? How is that unterminated? It’s got all three ‘#’ characters after the s, has the modifiers ‘i’ and ‘g’ after the third ‘#’, and the entire expression is in double quotes (‘”‘). Also, there are no escapes (‘\’) immediately preceding the delimiters, and the delimiter is not a part of either the search or the replacement. Let’s try a different delimiter than ‘#’, like ‘~’:

I use:
| sed -e “s~^(/[a-z]/.*/bin)$~A40-\1~ig”

and, I get:

    ...
    /c/si
    A40-/c/VS10/VC/bin
    /c/VS10/VC/bin/amd64
    /c/VS10/VC/bin/x86_amd64
    A40-/c/GnuWin32/bin
    ...

And, that is correct! The only thing I changed was the delimeter from ‘#’ to ‘~’ and it worked ???

This is not (even close to) the first time that sed has produced unexplainable results for me.

Why, oh, why, is sed NOT matching syntax in an expression in the same instance, but IS matching when piped into another instance of sed?
And, why, oh, why, do I have to use a different delimeter when I do this (in order not to get an “unterminated ‘s’ command”?

And the real reason I’m asking: Is this a bug in sed, OR, is it correct behavior that I don’t understand (and if so, can someone explain why this behavior is correct)? I want to know if I’m doing it wrong, or if I need a different/better tool (or both, they don’t have to be mutually exclusive).

I’ll mark a response it as the answer if someone can either prove why this behavior is correct or if they can prove why it is a bug. I’ll gladly accept any advice about other tools or different methods of using sed, but those won’t answer the question.

I’m going to have to get better at other text processors (like awk, tr, etc.) because sed is costing me too much time with it’s unexplainable results.

P.S. This is not the complete logic of my PATH manipulation. The complete logic also finishes prepending all the lines with values from ‘A00-‘ to ‘Z99-‘, then pipes that output into ‘sort -u -f’ and back into sed to remove those same prefixes on each line and to convert the lines (‘\n’) back into colons (‘:’). Then “export PATH='” is prepended to the single line and “‘” is appended to it. Then that output is redirected into a temporary file. Next, that temporary file is sourced. And, finally, that temporary file is removed.

The /etc/profile script also displays the contents of PATH before and after sorting (in case it screwed up the path).

P.P.S. I’m sure there is a much better way to do this. It started as some very simple sed manipulations, and grew into the monster you see here. Even if there is a better way, I still need to know why sed is giving me these results.

Answer 1

sed -e "s#^\(/[a-z]/.*/bin\)$#A40-\1#ig"

is unterminated because the shell is trying to expand “$#A”. Put your expressions in single quotes to avoid this.

The expression

-e "s#\n\(/[a-z]/.*/bin\)\n#\nA40-\1\n#ig"

fails, or doesn’t do what you expect, because . matches the newline in a multi-line expression. Check your whole output, the A40- is at the very beginning. Change it to

-e "s#\n\(/[a-z]/[^\n]*/bin\)\n#\nA40-\1\n#ig"

and it might be more what you expect. This may very well be the case with most of your issues with multi-line modifications.

You can also put the statements, one per line, into a standalone file and invoke sed with sed -f editscript. It might make maintenance of this a bit easier.