Question

I have a binary stream on standard input, it’s in a fixed size format, a continuos stream of packets, each packet has a header with length X and a body with length Y.

So if X=2 Y=6 then it’s something like 00abcdef01ghijkl02mnopqr03stuvwx, but it’s binary and both the header and data can contain any “characters” (including ‘\0’ and newline), the example is just for readability.

I want to get rid of the header data so the output looks like this: abcdefghijklmnopqrstuvwx.

Are there any commands in the unix toolchain that allow me to do this? And in general are there any tools for handling binary data? The only tool I could think of is od/hexdump but how do you convert the result back to binary?

Answer 1

Use xxd which goes to and from a hexdump.

xxd -c 123 -ps

will output your stream with 123 bytes per line. To reverse use

xxd -r -p

You should now be able to put this together with cut to drop characters since you can do something like

cut -c 3-

to get all characters from 3 to the end of a line. Do not forget to use a number of characters equal to 2X to account for two hex characters per byte.

So something along the lines of

xxd -c X+Y -ps | cut -c 2X+1- | xxd -r -p

where X+Y and 2X+1 are replaced with actual numerical values. You’ll need to put your datastream somewhere appropriate in to the above command.