Question

Say, I am writting some toolset where every single tool operates on the same textual data stream, parses it, does some operation on it and returns textual stream back using the same syntax as in the original input. The tools can be
combined (together with other unix tools/scripts/whatever) in a pipeline. Because the
textual input processing (parsing) is quite expensive, I would like to avoid it in case
two or more tools from the toolset are one right after another in the pipeline and use
binary streams instead (to store directly in a memory struct, w/o useless “extra” parsing). Is it
possible to know (using some trick, inter-process communication, or whatever else) if
the tool “before” or “after” any tool in a pipeline is part of the toolset? I guess the
unix env. is not prepared for such sort of “signalling” (AFAIK). Thanks for your ideas…

Answer 1

No, processes that are piped together have no methods of two-way communication. If the parsing is really so expensive that this is necessary (I’d guess it isn’t, but profile it), then you have a two options that I can think of:

• Have a master program that takes options to tell it which tools to run, in which order, and then have it run a “parse” tool, followed by the requested tools (all using binary I/O), followed by an “output” tool. It wouldn’t be terribly difficult to also expose the individual tools, wrapped with the parse/output tools.

• If users are expected to be knowledgeable enough, have each tool allow flags to tell them to expect binary input and give binary output, so that users can chain like:

  tool1 -o | tool2 -i -o | tool3 -i -o | tool4 -i
  

  where -o means give binary output and -i means accept binary input.