Question

Where is the buffer in this following … and how do I turn it off?

I am writing out to stdout in a python program like so:

for line in sys.stdin:
    print line

There is some buffering going on here:

tail -f data.txt | grep -e APL | python -u Interpret.py

I tried the following to shake off possible buffering … with no luck:

• as above using the -u flag with python invocation
• calling sys.stdout.flush() after each sys.stdout.write() call
  … all of these create a buffered stream with python waiting something like a minute to print out the first few lines.

• used the following modified command:

  stdbuf -o0 tail -f data.txt | stdbuf -o0 -i0 grep -e APL | stdbuf -i0 -o0 python -u Interpret.py

To benchmark my expectations, I tried:

tail -f data.txt | grep -e APL 

This produces a steady flow of lines … it surely is not as buffered as the python command.

So, how do I turn off buffering?
ANSWER: It turns out there is buffering on both ends of the pipe.

Answer 1

The problem, I believe is in grep buffering its output. It is doing that when you pipe tail -f | grep ... | some_other_prog. To get grep to flush once per line, use the --line-buffered option:

% tail -f data.txt | grep -e APL --line-buffered | test.py
APL

APL

APL

where test.py is:

import sys
for line in sys.stdin:
    print(line)

(Tested on linux, gnome-terminal.)