Question

I have a python subprocess call which I would like to link up to three pipes (two standard in and one standard out). I know that there is only one /dev/stdin, but there’s all those other devices in /dev I don’t know about, and don’t know of any python os, sys or subprocess modules that will utilise them in a manner which allows me to give the device path to subprocess.Popen.
The reason I ask is because I would like to pipe information from a mysql database or tar archive rather than a directory structure I currently have which has >28,000 directories in. The directory names alone uses a LOT of space! The alternative is to tar / gunzip the entire directory structure and manoeuvre through the compressed archive. With either solution, mysql or tar, I would still like to have two pipes into subprocess.Popen and one out, so that I can bypass the HDD.

Any need for an example??

Answer 1

On Unix systems, a convenient alternative is to use a named pipe. It looks like a file, but takes up no space on disk; you can write to it with one process and read from it with another, just like pipes. You can have your sub-process just do ordinary file I/O; Unix (Mac OS / Linux) will do the heavy lifting for you.

% mkfifo foo
% cat giantFile > foo &
[1] 4667
% wc -l foo
100