Question

Let’s say I have a ton of data flowing through stdout over a long period of time, maybe an hour, and I want to capture a 30 second window of that data based on a trigger that occurs in the middle of that window. For instance, maybe something like

$ program-that-outputs-lots-of-data | program-that-captures-a-window-of-data

At some point, a line that contains “A-unique-string” will be output by the program, and at that point I want to save the 15 seconds worth of data before and after that string, discarding everything before that. Immediately afterward, I want to start monitoring again for the same string and capture another window when it comes in and save it to a new file. Any idea how I can do something like this with Linux tools?

Answer 1

The fact that you are trying to use time as a unit for buffering makes your problem very rare. Under the Unix command line, everything tends to be designed around the text line concept.

For example, if instead of 15 seconds of data you would like to capture 15 lines of text (before and after the special token), you could simply do:

$ program-that-outputs-lots-of-data | grep -C 15 A-unique-string

In your case, even if you are developing your own tailored filtering tool, deciding how much input to save and to discard is a pretty complex problem. I’d say that multimedia streaming is the area where there might be some ready to use tools.