Question

I have an SQL Table named samples, defined like this:

sampledate (datetime, 24 records per day per parameter)
parameterID (int)
value (decimal)
valid (bit, 1=valid data, 0=invalid data)

the couple sampledate and parameterid are unique.

each sampledate is in the format 02/02/2011 12:00, so there are 24 rows per parameterid per day or less (a probe can fail or be in maintenance, for example, and it will output less than 24 samples).

I have to calculate the average daily values per parameter. The average is valid for a given day only if

1. at least 18 valid values are present
2. no more than 5 invalid consecutive values are present

Condition 1) is pretty simple to achieve, for a given @parameter:

  SELECT CONVERT(DATETIME, FLOOR(CONVERT(FLOAT, sampledate))) as avgdate,
         AVG(value) as avg, parameterID, 
         isValid = CASE  
           WHEN COUNT(value) > 17 THEN 1
           ELSE 0
         END 
    FROM samples
   WHERE parameterId=@parameter
GROUP BY parameterId, CONVERT(DATETIME, FLOOR(CONVERT(FLOAT, sampledate))), valid
  HAVING valid = 1  
ORDER BY sampledate

How can I add condition 2, which boils down to counting consecutive 0s in a 24hrs span, possibly with the best performances?

we have millions of samples, and cursors are slow.

Answer 1

And here goes my recursive CTE solution, and it is parameterisable:

WITH
seq_samples AS (
  SELECT
    sampledate, parameterID, value, valid,
    avgdate = CAST(FLOOR(CAST(sampledate AS float)) AS datetime),
    rownum = ROW_NUMBER() OVER (
      PARTITION BY parameterID, CAST(FLOOR(CAST(sampledate AS float)) AS datetime)
      ORDER BY sampledate)
  FROM samples
),
rec_samples AS (
  SELECT
    sampledate, parameterID, value, valid, avgdate, rownum,
    inv_seq_num = 1 - valid
  FROM seq_samples
  WHERE rownum = 1
  UNION ALL
  SELECT
    ss.sampledate, ss.parameterID, ss.value, ss.valid, ss.avgdate, ss.rownum,
    inv_seq_num = CASE ss.valid WHEN 1 THEN 0 ELSE rs.inv_seq_num + 1 END
  FROM seq_samples ss
    INNER JOIN rec_samples rs ON ss.avgdate = rs.avgdate
      AND ss.parameterID = rs.parameterID AND ss.rownum = rs.rownum + 1
)
SELECT
  avgdate,
  parameterID,
  avgvalue = AVG(value)
FROM rec_samples
GROUP BY avgdate, parameterID
HAVING SUM(CAST(valid AS int)) >= @minimal_valid_count
   AND MAX(inv_seq_num)        <= @critical_invalid_count

Your idea basically is implemented here. Additional numbering is used, which is only applied to invalid rows and is only broken by date transitions and valid values’ occurrences. In the end MAX is applied to the numbering column to find out if the max number has not exceeded @critical_invalid_count. And for the other parameter it is obviously enough to check the sum of valid attributes.

So, there you are.

---

EDIT for the seq_samples CTE (to be applied to your adapted version of the original query).

seq_samples AS (
  SELECT
    *,
    rownum = ROW_NUMBER() OVER (
      PARTITION BY parameterID, avgdate
      ORDER BY sampledate)
  FROM (
    SELECT
      sampledate, parameterID, value, valid,
      avgdate = CAST(FLOOR(CAST(sampledate AS float)) AS datetime)
    FROM samples
  ) s
),

SSMS showed me significant, practically unbelievable difference in performance between my original query and the modified one. (This is only based on the figures from the estimated execution plan.) I don’t know what adaptations you had to make to my original solution, but I hope the improvement I’ve witnessed will not be completely lost because of them.