Question

I have an OLTP database, and am currently creating a data warehouse. There is a dimension table in the DW (DimStudents) that contains student data such as address details, email, notification settings.

In the OLTP database, this data is spread across several tables (as it is a standard OLTP database in 3rd normal form).

There are currently 10,390 records but this figure is expected to grow.

I want to use Type 2 ETL whereby if a record has changed in the OLTP database, a new record is added to the DW.

What is the best way to scan through 10,000 records in the DW and then compare the results with the results in several tables contained in the OLTP?

I’m thinking of creating a “snapshot” using a temporary table of the OLTP data and then comparing the results row by row with the data in the Dimension table in the DW.

I’m using SQL Server 2005. This doesn’t seem like the most efficient way. Are there alternatives?

Answer 1

• Introduce LastUpdated into source system (OLTP) tables. This way you have less to extract using:

  WHERE LastUpdated >= some_time_here

You seem to be using SQL server, so you may also try rowversion type (8 byte db-scope-unique counter)

• When importing your data into the DW, use ETL tool (SSIS, Pentaho, Talend). They all have a componenet (block, transformation) to handle SCD2 (slowly changing dimension type 2). For SSIS example see here. The transformation does exactly what you are trying to do — all that you have to do is specify which columns to monitor and what to do when it detects the change.