Question

I have a (java) program that prints a line of hex numbers to stdout every 5ish seconds, until the program is terminated by the user.

I would like to redirect that output to a bash script so I could convert each of those hex numbers independently to decimal, then print the parsed line to stdout.

I tried using myProgram | myScript but that did the piping before any lines were printed, then didn’t keep listening to stdout. I then tried myProgram > myScript, and that just overwrote the script.

Ideas?

Edit: adding output from the runs, (sorry for the poor formatting, I couldn’t get it all in the code highlighting) so the middle of the output is not highighted).

Here is the script

#!/bin/bash
echo $0
echo $#
echo $1

Here is how my program runs while it goes straight to stdout this would continue forever if I didn’t terminate it.

mmmm@mmmm:~/mmmm/mmmm/mmmmm$ java net.tinyos.tools.Listen -comm 
serial@/dev/ttyUSB0:micaz
serial@/dev/ttyUSB0:57600: resynchronising   
00 FF FF 00 02 04 22 93 00 02 02 C9
00 FF FF 00 03 04 22 93 00 03 03 0E
00 FF FF 00 02 04 22 93 00 03 03 0E
00 FF FF 00 02 04 22 93 00 02 02 C9
^Z
[5]+  Stopped                 java net.tinyos.tools.Listen -comm          
serial@/dev/ttyUSB0:micaz

Here is where I try to pipe it to my script (which i have set to print the number of command line arguments and the first argument. It just freeze after this…

mmmm@mmmm:~/mmmm/mmmm/mmmmm$$ java net.tinyos.tools.Listen -comm serial@/dev/ttyUSB0:micaz | ./parser.sh
./parser.sh
0
serial@/dev/ttyUSB0:57600: resynchronising

Answer 1

Diagnosis

When you use this script like this:

java javaprog | myScript

and myScript contains:

#!/bin/bash
echo $0
echo $#
echo $1

Then the output from the script will be its name (myScript) from the echo $0, the number of arguments it was passed (0) from the echo $#, and the first argument (an empty line is echoed) from the echo $1. The script then exits (successfully). The issue is nothing to do with buffering; it is all to do with the script not reading anything from its standard input. Even a trivial modification would be an improvement:

#!/bin/bash
while read data; do echo $data; done

That’s a slower form of cat, except that it normalizes random sequences of spaces and tabs into single spaces, stripping leading and trailing spaces off the line. It would at least demonstrate the script processing the output from the Java program.

---

Trying awk

To do what you’re after, you should probably replace that with an awk program or something similar. This is a first draft, but it stands some chance of working:

awk '{for(i = 1; i <= NF; i++) { x = "0x" $i + 0; printf(" %d", x); printf "\n";}'

This says ‘for each line (because there is no pattern before the open brace)’, do ‘for each of the fields 1..NF, convert the field into an explicit hex string with the 0x prefix and adding 0, then print the value as a decimal number (trusting awk to convert a string such as ‘0xC9’ to a number).

Using Perl

Unfortunately, a little testing shows that this does not work; the problem is getting a value other than 0 for x. So, … time to fall back on Perl in awk-emulation mode:

$ echo '00 C9 28 13 A0 FF 01' |
> perl -na -e 'for ($i = 0; $i < scalar(@F); $i++) { printf(" %d", hex $F[$i]); }
>       printf "\n";'
 0 201 40 19 160 255 1
$

That works – it’s even fairly easy to understand. The -n option means ‘read each line of data and execute the commands in the script on each line (but do not print $_ at the end)’. The -a option combined with either -n (as here, or -p which is like -n except it prints $_ automatically) means ‘automatically split the input into the array @F. The script then processes each element of @F in each line (rather verbosely), using the hex function to convert the string in $F[$i] to a number and then printing that number with printf(). The verbosity can be reduced (this is Perl: There’s More Than One Way To Do It, or TMTOWTDI – tim-toady) with:

$ echo '00 C9 28 13 A0 FF 01' |
> perl -na -e 'foreach my $i (@F) { printf(" %d", hex $i); } printf "\n";'
 0 201 40 19 160 255 1
$

Same result, less code. There might be more abbreviated techniques; that’s compact enough without being wholly illegible.