Question

This is similar to Compute dates and durations in mysql query, except that I don’t have a unique ID column to work with, and I have samples not start/end points.

As an interesting experiment, I set cron to ps aux > 'date +%Y-%m-%d_%H-%M'.txt. I now have around 250,000 samples of “what the machine was running”.

I would like to turn this into a list of “process | cmd | start | stop”. The assumption is that a ‘start’ event is the first time when the pair existed, and a ‘stop’ event is the first sample where it stopped existing: there is no chance of a sample “missing” or anything.

That said, what ways exist for doing this transformation, preferably using SQL (on the grounds that I like SQL, and this seems like a nice challenge). Assuming that pids cannot be repeated this is a trivial task (put everything in a table, SELECT MIN(time), MAX(time), pid GROUP BY pid). However, since PID/cmd pairs are repeated (I checked, there are duplicates), I need a method that does a true “find all contiguous segments” search.

If necessary I can do something of the form

Load file0 -> oldList
ForEach fileN:
    Load fileN ->newList
    oldList-newList = closedN
    newList-oldList = openedN
    oldList=newList

But that is not SQL and not interesting. And who knows, I might end up having real SQL data to deal with with this property at some point.

I’m thinking something where one first constructs a table of diff’s, and then joins all close’s against all open’s and pulls the minimum-distance close after each open, but I’m wondering if there’s a better way.

Answer 1

You don’t mention what database you are using. Let me assume that you are using a database that supports ranking functions, since that simplifies the solution.

The key to solving this is an observation. You want to assign an id to each pid to see if it is unique. I am going to assume that a pid represents a single process when the pid did not appear in the previous timestamped output.

Now, the idea is:

1. Assign a sequential number to each set of output. The first call to ps gets 1, the next 2, and so on, based on date.
2. Assign a sequential number to each pid, based on date. The first appearance gets 1, the next 2, and so on.
3. For pids that appear in sequence, the difference is a constant. We can call this the groupid for that set.

So, this is the query in action:

select groupid, pid, min(time), max(time)
from (select t.*,
             (dense_rank() over (order by time) -
              row_number() over (partition by pid order by time)
             ) as groupid
      from t
     ) t
group by groupid, pid

This works in most databases (SQL Server, Oracle, DB2, Postgres, Teradata, among others). It does not work in MySQL because MySQL does not support the window/analytic functions.