Question

I wrote a piped shell command that has multiple pipes in it that works great. I now want to put this in the form of a (tidy) shell script. Here is the script:

#!/bin/bash
for number in `cat xmlEventLog_2010-03-23T* | sed -nr "/<event eventTimestamp/,/<\/event>/ {/event /{s/^.*$/\n/; p};/payloadType / {h; /protocol/ {s/.*protocol=\"([^\"]*)?\".*/protocol: \1/}; p; x; /type/ {s/.*type=\"([^\"]+)\".*/payload: \1/g}; /type/! {s/.*protocol=\"([^\"]+)\".*/payload: \1/g}; p};/sender / {/sccpAddress/ {s/.*sccpAddress=\"([^\"]*)?\".*/sccpAddress: \1/}; /sccpAddress/! {s/.*/sccpAddress: Unknown/}; p};/result /{s/.*value=\"([^\"]+)\".*/result: \1/g; p};/filter code/{s/.*type=\"([^\"]+)\".*/type: \1/g; p};}"| tee checkThis.txt| awk 'BEGIN{FS="\n"; RS=""; OFS=";"; ORS="\n"} $1~/result: Blocked|Modified/ && $2~/sccpAddress: 353201000001/ && $4~/payload: SMS-MO-FSM-INFO|SMS-MO-FSM/ {$1=$1 ""; print}' | sort | uniq -c| egrep "NUMBER_BLACKLIST|USER_BLACKLIST|NUMBER_WALLEDGARDEN|USER_WALLED_GARDEN|SERVICE_RESTRICTION|BLOCK_VOICE_TO_SMS|PEP_Blacklist_Whitelist" | awk '{print $1}'`; do fil="$fil+$number"
done
echo "fil is $fil"

I would like to tidy this up so that it is readable. The for loop which pipes into sed and awk is pig ugly to view. Has anybody got suggestions to tidy up this piped monstrosity. Would the pipes stop me from breaking this up onto different lines?

Thanks

A

If you copy the lines above to notepad you will see what I mean about ugly (but functional)

Ok folks. Here is the final cleaned up version.

It was mentioned that the event_structure function could be done entirely in awk. I wonder if anybody could show me an example of how this could be done. The record separator would be set to /event and that would separate the events but it’s the structures that are in events.txt (see below) that I’m interested in. The number outcome is immaterial.

The core of the code is in the event_structure function. I want to parse out the data and put it all into data structures for later inspection should the case arrise. The following works fine. On the line that starts with payloadType I need to parse out 2 values or set any missing values to Unknown. Is this totally awkable or is the sed/awk combination I have here the best way to do this?

#!/bin/bash

event_structure() {
      sed -nr "/<event eventTimestamp/,/<\/event>/ {
            /event /{s/^.*$/\n/; p}
            /payloadType / {h; /protocol/ {s/.*protocol=\"([^\"]*)?\".*/protocol: \1/}; p; x; /type/ {s/.*type=\"([^\"]+)\".*/payload: \1/g}; /type/! {s/.*protocol=\"([^\"]+)\".*/payload: \1/g}; p}
            /sender / {/sccpAddress/ {s/.*sccpAddress=\"([^\"]*)?\".*/sccpAddress: \1/}; /sccpAddress/! {s/.*/sccpAddress: Unknown/}; p}
            /result /{s/.*value=\"([^\"]+)\".*/result: \1/g; p}
            /filter code/{s/.*type=\"([^\"]+)\".*/type: \1/g; p};}" xmlEventLog_2010-03-23T* |
      tee events.txt|
      awk 'BEGIN{FS="\n"; RS=""; OFS=";"; ORS="\n"}
      $1~/result: Blocked|Modified/ && $2~/sccpAddress: 353201000001/ && $4~/payload: SMS-MO-FSM-INFO|SMS-MO-FSM/ {$1=$1 ""; print}'
}

numbers=$(event_structure | sort | uniq -c | egrep "NUMBER_BLACKLIST|USER_BLACKLIST|NUMBER_WALLEDGARDEN|USER_WALLED_GARDEN|SERVICE_RESTRICTION|BLOCK_VOICE_TO_SMS|PEP_Blacklist_Whitelist" | awk '{print $1}')
addition=`echo $numbers | tr -s ' \n\t' '+' | sed -e '1s/^/fil is /' -e '$s/+$//'`
for number in $numbers
do
      fil="$fil+$number"
done
echo $addition=$(($fil))

Here is a section of the events.txt file produced:

result: Blocked
sccpAddress: 353869000000
protocol: SMS
payload: COPS
type: SERVICE_BLACK_LIST
result: Blocked


result: Blocked
sccpAddress: 353869000000
protocol: SMS
payload: COPS
type: SERVICE_BLACK_LIST
result: Blocked

result: Modified
sccpAddress: Unknown
protocol: IM
payload: IM
type: NUMBER_BLACKLIST
result: Modified

result: Allowed
sccpAddress: Unknown
protocol: MM1
payload: MM1

Here is the output:

$ ./bashShell.sh
fil is 2+372+1+1+214+73+1+20=684

Here is an output of just the function call:

$ ./bashShell.sh | head -10
result: Blocked;sccpAddress: 353201000001;protocol: SMS;payload: SMS-MO-FSM;type: TEXT_ANALYSIS;result: Blocked
result: Blocked;sccpAddress: 353201000002;protocol: SMS;payload: SMS-MT-FSM;type: TEXT_ANALYSIS;result: Blocked
result: Blocked;sccpAddress: 353201000005;protocol: SMS;payload: SMS-MO-FSM;type: SERVICE_BLACKLIST;result: Blocked
result: Blocked;sccpAddress: 353201000021;protocol: SMS;payload: SMS-MT-FSM;type: NUMBER_BLACKLIST;result: Blocked
result: Blocked;sccpAddress: 353201000033;protocol: IM;payload: IM;type: NUMBER_BLACKLIST;result: Blocked
result: Blocked;sccpAddress: 353401009001;protocol: SMS;payload: SMS-MO-FSM;type: NUMBER_BLACKLIST;result: Blocked
result: Blocked;sccpAddress: 353201000001;protocol: SMS;payload: SMS-MO-FSM;type: NUMBER_BLACKLIST;result: Blocked
result: Blocked;sccpAddress: 353201000005;protocol: SMS;payload: SMS-MO-FSM;type: NUMBER_BLACKLIST;result: Blocked
result: Blocked;sccpAddress: 353401000001;protocol: SMS;payload: SMS-MO-FSM;type: NUMBER_BLACKLIST;result: Blocked
result: Blocked;sccpAddress: 353201000001;protocol: SMS;payload: SMS-MO-FSM;type: NUMBER_BLACKLIST;result: Blocked

p.s I named the script bashShell.sh for no particular reason

A

Answer 1

Pipes don’t stop you when breaking to multiple lines, but use $( ... ) instead of backticks. Something like this should work:

#!/bin/bash

for number in $(
    cat xmlEventLog_2010-03-23T* |
    sed -nr "/<event eventTimestamp/,/<\/event>/ {/event /{s/^.*$/\n/; p};/payloadType / {h; /protocol/ {s/.*protocol=\"([^\"]*)?\".*/protocol: \1/}; p; x; /type/ {s/.*type=\"([^\"]+)\".*/payload: \1/g}; /type/! {s/.*protocol=\"([^\"]+)\".*/payload: \1/g}; p};/sender / {/sccpAddress/ {s/.*sccpAddress=\"([^\"]*)?\".*/sccpAddress: \1/}; /sccpAddress/! {s/.*/sccpAddress: Unknown/}; p};/result /{s/.*value=\"([^\"]+)\".*/result: \1/g; p};/filter code/{s/.*type=\"([^\"]+)\".*/type: \1/g; p};}"|
    tee checkThis.txt |
    awk 'BEGIN{FS="\n"; RS=""; OFS=";"; ORS="\n"} $1~/result: Blocked|Modified/ && $2~/sccpAddress: 353201000001/ && $4~/payload: SMS-MO-FSM-INFO|SMS-MO-FSM/ {$1=$1 ""; print}' |
    sort |
    uniq -c |
    egrep "NUMBER_BLACKLIST|USER_BLACKLIST|NUMBER_WALLEDGARDEN|USER_WALLED_GARDEN|SERVICE_RESTRICTION|BLOCK_VOICE_TO_SMS|PEP_Blacklist_Whitelist" |
    awk '{print $1}'
  ); do fil="$fil+$number"
done
echo "fil is $fil"

Of course the larger part is to split the awk and sed skripts into multiple lines also…

But I believe that even after that the result will be still quite unreadable.

I would suggest just completely rewriting the script in Perl, Ruby or any other a bit more readable scripting language than Bash. This is just a suggestion from my personal experience – every time a start out with a shell script I finally rewrite it in Ruby. I love Bash, but it just doesn’t seem to scale.