Home/ Questions/Q 972943
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-16T03:12:18+00:00

I have a system which records some measured values every second. What is the

  • 0

I have a system which records some measured values every second. What is the best way to store trend data which are values corresponding to a specific second?

1 day = 86.400 seconds
1 month = 2.592.000 seconds

Around 1000 values to keep track of every seconds.

Currently there are 50 tables grouping the trend data for 20 columns each. These tables contain more than 100 million rows.

    TREND_TIME datetime (clustered_index)
    TREND_DATA1 real
    TREND_DATA2 real
    ...
    TREND_DATA20 real
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    I would change the data saving approach and instead of saving ‘raw’ data as values I would save 5-20 minutes of data in an array (Memory, BL side), compress that array using LZ based algorithm and then store the data in the database as binary data. Also, it would be nice to save Max/Min/Avg/etc.. info for that binary chunk.

    When you want to process the data you can process the data chunk after chunk and by that you keep a low memory profile for your application. this approach is a little more complex but very scalable in terms of memory/processing.

    hope this helps.

    • 0