Question

I want to capture output from two concurrent programs (tails on logfiles) into one output stream in bash.

I used this example program for testing:

function foo { for i in $(seq 1 10); do echo "program $*"; sleep 1; done }

Now the following works fine

(foo bar & foo baz &) | tee /tmp/output

but once I add an extra pipe in the mix it no longer works:

(foo bar | grep bar & foo baz &) | tee /tmp/output # does't work

The output becomes sequential. I could make a separate program that includes the grep but I’d like to know if there is a way around this.

If someone can explain why it doesn’t work I would be very happy.

Answer 1

Great question! This one had me stumped for a bit but I think I know what’s going on. What’s happening is that grep is buffering the output. So, if you let it run you’ll see it all flood at the end. If you happen to be using GNU grep try passing the –line-buffered option:

(foo bar | grep --line-buffered bar & foo baz &) | tee /tmp/output

To hazard a guess, and mind you that’s essentially what it is, I’d say that grep is buffering more output because isatty(1) would indicate that it’s not writing to a TTY (even though you are watching the output on a TTY via tee). By buffering more output it makes fewer write() calls, and is more efficient. The familiar behavior of running grep and watching the output in a terminal is line buffered — lines appear as they’re found. This option forces grep to run in that mode.

Keep in mind, as the man page warns, this could have performance impacts on grep.