Question

I have a MySQL table like this:

CREATE TABLE IF NOT EXISTS `mytable` (
  `ts` datetime NOT NULL,
  `cnt` int(10) unsigned NOT NULL,
  PRIMARY KEY (`ts`)
) ENGINE=InnoDB;

where we store the value of an event counter whenever we get an update; those updates arrive at arbitrary time.

How could I extract the number of events for every X amount of time (eg. 5 minutes, a day, a month, etc)? I could simplify this to intervals easily extracted via date/time sql functions (eg. hour, day, month, etc…).

While we don’t have any guarantee, the average data is “dense” when compared to the intervals I’d like to extract. EG. data usually comes in multiple times every hour but I’ll never ask for the count of events in an interval < 1 hour. If there is a “problem” (eg. big holes) in the stored data, it is acceptable to have a “problem” in the results.

As an example, I could get the counter values I’m interested in with a query like this (24h period example):

SELECT ts, cnt
FROM mytable
GROUP BY DATE( ts ) 
ORDER BY ts DESC

…and the events count could easily be calculated by subtracting each row’s counter with its predecessor. But I’d like to do that in SQL, if possible.

Also, if there is a good name for this problem (I think it’s a rather common one when you work with time series and counters) I’d like to know it to improve my vocabulary 🙂

Answer 1

If you would add an AUTO_INCREMENT PRIMARY KEY onto your table, that would be aolveable.

The table schema as you have presented it is invalid (PRIMARY KEY on timestamp, but no such column).

Would you mind if we:

ALTER TABLE mytable DROP PRIMARY KEY, ADD COLUMN id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY

?

If so, then the following could be done, I’ll present in steps:

SELECT
  TIMESTAMPDIFF(SECOND, m1.ts, m2.ts) AS diff_seconds,
  m2.cnt - m1.cnt AS diff_cnt
FROM
  mytable m1 JOIN mytable m2 ON (m1.id = m2.id-1)
;

The above shows the difference in time and in cnt between consecutive samples. Let’s add a third column:

SELECT
  TIMESTAMPDIFF(SECOND, m1.ts, m2.ts) AS diff_seconds,
  m2.cnt - m1.cnt AS diff_cnt,
  (m2.cnt - m1.cnt)/TIMESTAMPDIFF(SECOND, m1.ts, m2.ts) AS cnt_per_second
FROM
  mytable m1 JOIN mytable m2 ON (m1.id = m2.id-1)
;

I evaluated cnt_per_second. Multiply by 60 to get cnt per minute, and so on.

Now, the total average would be:

SELECT 
  avg(cnt_per_second)
FROM (
  SELECT
    TIMESTAMPDIFF(SECOND, m1.ts, m2.ts) AS diff_seconds,
    m2.cnt - m1.cnt AS diff_cnt,
    (m2.cnt - m1.cnt)/TIMESTAMPDIFF(SECOND, m1.ts, m2.ts) AS cnt_per_second
  FROM
    mytable m1 JOIN mytable m2 ON (m1.id = m2.id-1)
) sel_diff
;

Add original n1.ts to first query if you want to know when a diff was recorded, and so you will also be able to know the average count events in a given time period.