Question

I’m writing a shell script that splits line of string based on a pattern using sed.

    #pattern 'string1','string2','string3'

    cat $FILENAME | while read LINE
    do

    firstPart=$(echo "$LINE" | sed -r "s/'(.*)','(.*)','(.*)'/\1/" )
    secondPart=$(echo "$LINE" | sed -r "s/'(.*)','(.*)','(.*)'/\2/" )
    thirdPart=$(echo "$LINE" | sed -r "s/'(.*)','(.*)','(.*)'/\3/" )

    done

I am able to print them using individual echos, but if I put them in a single echo as show below

    #if LINE from FILE is '123','abc','hello'

    echo "$firstPart $secondPart"
    #this prints " abc" instead of "123 abc"

    #try appending a string on echo
    echo "$firstPart -"
    #this prints " -3" instead of "123 -"

When i tried using sed in a constant string in the code, echo seems fine.

    #Correct Echo
    SOMESTRING='123','abc','hello'

    firstPart=$(echo "$SOMESTRING" | sed -r "s/'(.*)','(.*)','(.*)'/\1/" )
    secondPart=$(echo "$SOMESTRING" | sed -r "s/'(.*)','(.*)','(.*)'/\2/" )
    thirdPart=$(echo "$SOMESTRING" | sed -r "s/'(.*)','(.*)','(.*)'/\3/" )

    echo "$firstPart $secondPart"
    #this prints "123 abc"

Is it the correct behavior of sed when input is a LINE from FILE? how can i make it behave as if the LINE is included and declared in the code(like my second example).

Answer 1

It looks to me like you have carriage returns (sometimes written \r) embedded in your strings. So when you do echo "$firstPart -", where firstPart=”123\r”, it prints two lines:

123
 -

…with the second “line” printed on top of (and overwriting) the first. My guess is that you’ve edited the script with a text editor that uses DOS-style line endings (i.e. each line ends with carriage return followed by linefeed), but the shell expects unix-style line endings (just linefeed) and treats the carriage return as part of the command (e.g. firstPart=$(echo "$SOMESTRING" | sed -r "s/'(.*)','(.*)','(.*)'/\1/" )\r, which will include the \r in firstPart.

If this is the problem, running dos2unix on your script should fix it (and then switch to an editor that doesn’t use DOS-style line endings).