Question

I have two programs A and B. I can’t change the program A – I can only run it with some parameters, but I have written the B myself, and I can modify it the way I like.

Program A runs for a long time (20-40 hours) and during that time it produces output to the file, so that its size increases constantly and can be huge at the end of run (like 100-200 GB). The program B then reads the file and calculates some stuff. The special property of the file is that its content is not correlated: I can divide the file in half and run calculations on each part independently, so that I don’t need to store all the data at once: I can calculate on the first part, then throw it away, calculate on the second one, etc.

The problem is that I don’t have enough space to store such a big files. I wonder if it is possible to pipe somehow the output of the A to B without storing all the data at once and without making huge files. Is it possible to do something like that?

Thank you in advance, this is crucial for me now, Roman.

Answer 1

If program A supports it, simply pipe.

A | B

Otherwise, use a fifo.

mkfifo /tmp/fifo
ls -la > /tmp/fifo &
cat /tmp/fifo

EDIT: Adjust buffer sizes with ulimit -p and then:

cat /tmp/fifo | B