Home/ Questions/Q 610113
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-13T17:38:24+00:00

In the process industry, lots of data is read, often at a high frequency,

  • 0

In the process industry, lots of data is read, often at a high frequency, from several different data sources, such as NIR instruments as well as common instruments for pH, temperature, and pressure measurements. This data is often stored in a process historian, usually for a long time.

Due to this, process historians have different requirements than relational databases. Most queries to a process historian require either time stamps or time ranges to operate on, as well as a set of variables of interest.

Frequent and many INSERT, many SELECT, few or no UPDATE, almost no DELETE.

Q1. Is relational databases a good backend for a process historian?

A very naive implementation of a process historian in SQL could be something like this.

+------------------------------------------------+
| Variable                                       |
+------------------------------------------------+
| Id : integer primary key                       |
| Name : nvarchar(32)                            |
+------------------------------------------------+

+------------------------------------------------+
| Data                                           |
+------------------------------------------------+
| Id : integer primary key                       |
| Time : datetime                                |
| VariableId : integer foreign key (Variable.Id) |
| Value : float                                  |
+------------------------------------------------+

This structure is very simple, but probably slow for normal process historian operations, as it lacks “sufficient” indexes.

But for example if the Variable table would consist of 1.000 rows (rather optimistic number), and data for all these 1.000 variables would be sampled once per minute (also an optimistic number) then the Data table would grow with 1.440.000 rows per day. Lets continue the example, estimate that each row would take about 16 bytes, which gives roughly 23 megabytes per day, not counting additional space for indexes and other overhead.

23 megabytes as such perhaps isn’t that much but keep in mind that numbers of variables and samples in the example were optimistic and that the system will need to be operational 24/7/365.

Of course, archiving and compression comes to mind.

Q2. Is there a better way to accomplish this? Perhaps using some other table structure?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    It sounds like you’re talking about telemetry data (time stamps, data points).

    We don’t use SQL databases for this (although we do use SQL databases to organize it); instead, we use binary streaming files to capture the actual data. There are a number of binary file formats that are suitable for this, including HDF5 and CDF. The file format we use here is a proprietary compressible format. But then, we deal with hundreds of megabytes of telemetry data in one go.

    You might find this article interesting (links directly to Microsoft Word document):
    http://www.microsoft.com/caseStudies/ServeFileResource.aspx?4000003362

    It is a case study from the McClaren group, describing how SQL Server 2008 is used to capture and process telemetry data from formula one race cars. Note that they don’t actually store the telemetry data in the database; instead, it is stored in the file system, and the FILESTREAM capability of SQL Server 2008 is used to access it.

    • 0