Question

In bash, is there a way to chain multiple commands, all taking the same input from stdin? That is, one command reads stdin, does some processing, writes the output to a file. The next command in the chain gets the same input as what the first command got. And so on.

For example, consider a large text file to be split into multiple files by filtering the content. Something like this:

cat food_expenses.txt | grep "coffee" > coffee.txt | grep "tea" > tea.txt | grep "honey cake" > cake.txt

This obviously does not work, because the second grep gets the first grep’s output, not the original text file. I tried inserting tee’s but that does not help. Is there some bash magic that can cause the first grep to send its input to the pipe, not the output?

And by the way, splitting a file was a simple example. Consider splitting (filering by pattern search) a continuous live text stream coming over a network and writing the output to different named pipes or sockets. I would like to know if there is an easy way to do it using a shell script.

(This question is a cleaned up version of my earlier one , based on responses that pointed out the unclearness)

Answer 1

I like Stephen’s idea of using awk instead of grep.

It ain’t pretty, but here’s a command that uses output redirection to keep all data flowing through stdout:

cat food.txt | 
awk '/coffee/ {print $0 > "/dev/stderr"} {print $0}' 
    2> coffee.txt | 
awk '/tea/ {print $0 > "/dev/stderr"} {print $0}' 
    2> tea.txt

As you can see, it uses awk to send all lines matching ‘coffee’ to stderr, and all lines regardless of content to stdout. Then stderr is fed to a file, and the process repeats with ‘tea’.

If you wanted to filter out content at each step, you might use this:

cat food.txt | 
awk '/coffee/ {print $0 > "/dev/stderr"} $0 !~ /coffee/ {print $0}' 
    2> coffee.txt | 
awk '/tea/ {print $0 > "/dev/stderr"} $0 !~ /tea/ {print $0}' 
    2> tea.txt