Question

I’m having little problem with bash/sed. I need to be able to use command substitution within sed expression. I have two big text files:

• first is logfile.txt which sometimes* shows error messages by ID (0xdeadbeef is common example) in format ERRORID:0xdeadbeef

• second errors.txt has error messages stored in pairs LONG_ERROR_DESCRIPTION, 0xdeadbeef

I was trying to use sed with bash command substitution to do the task:

cat logfile.txt | sed "s/ERRORID:\(0x[0-9a-f]*\)/ERROR:$(cat errors.txt |
    grep \1 | grep -o '^[A-Z_]*' )/g"

(^^^ this should be in one line of course)

If it would work then I could get little nicer version of logfile with better error info.

   Lot's of meaningless stuff ERRORID:0xdeadbeef and something else =>
=> Lot's of meaningless stuff ERROR:LONG_ERROR_DESCRIPTION and something else 

But it doesn’t. The problem is that sed is unable to “inject” regex section (\1) into command substitution. What are my other options? I know that it’s possible to build sed expression first or do it other way but I would like to avoid parsing those files several times (they can be huge).

As always big thanks for any help.

*there is no real formatting inside logfile. No sections, columns, tab/coma-separation are used inconsistently

PS. Just to explain. Following expression works, but of course there is no argument passing within it:

echo "my cute cat" | sed "s/cat/$(echo dog)/g"

Answer 1

You can create a sed script from the error message catalog, then apply that sed script to the log file.

Basically, something along these lines:

sed 's/\(.*\), 0x\([0-9A-F]*\)$/s%ERRORID:0x\2%ERROR:\1%g/' errors.txt |
sed -f - logfile.txt

The output from the first sed script should be something like this:

s%ERRORID:0x00000001%ERROR:Out of memory%
s%ERRORID:0x00000002%ERROR:Stack overflow%
s%ERRORID:0x00000031%ERROR:values of beta may cause dom%

That is, a new sed script which specifies a substitution for each error code in the catalog.

There are different dialects of sed so this may require minor tweaking. The sed on Linux I believe should use backslash before grouping parentheses in regular expressions, and gladly tolerate standard input as the argument to the -f option. This is not portable to other Unices, though (but you could substitute Perl for sed if you need portability).

*Edit: If the error messages are fairly static, and/or you want to read the log from standard input, save the generated script in a file;

# Do this once
sed 's/\(.*\), 0x\([0-9A-F]*\)$/s%ERRORID:0x\2%ERROR:\1%g/' errors.txt >errors.sed
# Use it many times
sed -f errors.sed logfile.txt

You could also add #!/usr/bin/sed -f at the top of errors.sed and chmod +x it to make it into a self-contained command script.