I’ve worked on several apps and talked to other developers who have had problems with several details of data warehousing.
The main issue I’ve seen is regarding Change Data Detection (CDC) in the operational data store. Updates and hard deletes obviously can be hard to detect in the operational data store.
Updates can be handled by inserting triggers on EVERY table that automatically update the updated_at column with the current timestamp. Deletes are harder though – one solution is to put a trigger in that updates an audit table with the id deleted, the table, and a timestamp.
Using triggers seems like the most reasonable way to do change data detection, but another option I’ve seen is to parse the database transaction log files, though that may make it harder to update the operational data store database.
My question is, how do people usually handle this issue? I’ve done a fair bit of research and it really seems like a lot of companies doing data warehousing are rolling their own sub-optimal solutions.
Another solution I’ve seen to avoid problems related to CDC is to simply rebuild the ENTIRE (or the portion related to the source data) data warehouse every once in a while, which will ensure that all the data is current and there is no bug in the code that does CDC on the operational data store.
Here’s the way I typically handle updates and deletes.
Updates in the Source System
Some DBMS’s provide a column, which, if added to all the tables, provides the warehouse with a unique identifier that is always increasing. SQL Server has the TIMESTAMP column. Oracle provides the ora_rowscn pseudocolumn, which is good at the block level.
While I haven’t used it, Postgres has the xmin pseudocolumn, which I believe could be used in a similar fashion. There are some concerns over it but I think for data warehouse change tracking purposes, it may do the trick.
UPDATE triggers in the source system to update the last modified date are another option. Keep this date at a very high precision to reduce the risk of “missing” records if something is running doing massive updates on the ODS when you do your extract.
Deletes in the Source System
As for deleted records, my preferred solution is to ensure that all of the source tables have a primary key (preferably one column, though multiple is doable). I extract the entirety of this column into a stage table daily, then identify the rows that are “missing” from the target table compared to the source, an update a “source deleted” flag or something on the target records. I typically only do this for dimension tables, since fact tables should retain history even if the original transaction is gone.