Question

I have a file test.log. Very big log file. It has different levels of logging. For eg, trace, debug, info, warning and error.

Obviously trace level messages are just spamming at high speeds. I wanted to see all the messages without trace level logs.

So I did this:

cat test.log | grep -v "trace"

Works good.

Now I want to filter the remaining messages based on a certain keyword keyword1.

So I did this:

cat test.log | grep -v "trace" | grep "keyword1"

Works good.

Now I want to get that same output on a continual basis, I thought of replacing cat with tail -f.

tail -f test.log | grep -v "trace" | grep "keyword1"

But this does not work. I get no output at all.

What am I doing wrong? And how can I get my desired filtered ‘tail & follow’ output.

Thanks for the help.

(btw, I am using cygwin… if that matters in any way)

Answer 1

You are encountering buffering issues: for performance reasons grep will keep quite a bit of output in its buffer and will output the whole chunk in one go. That means that when a line is read from the input, grep will send it to stdout once it has read (and let through) even more lines – which can be quite some time later when dealing with a log file that is being read-in using tail -f.

Many grep variants have a switch to turn-on line-buffered mode, which will output each line on its own – with some performance loss. For example GNU grep has a --line-buffered option to achieve this effect.

Just add that option (or the appropriate one for your version of grep) to all your grep invocations and you see some output as soon as a matching like is added to the log file.