Question

Let’s say I’m getting a large (2 million rows?) amount of data that’s supposed to be static and unchanging. Supposed to be. And this data gets republished monthly. What methods are available to 1) be aware of what data points have changed from month to month and 2) consume the data given a point in time?

Solution 1) Naively save every snapshot of data, annotated by date. Diff awareness is handled by some in-house program, but consumption of the data by date is trivial. Cons, space requirements balloon by an order of magnitude.

Solution 2A) Using an in-house program, track when the diffs happen and store them in an EAV table, annotated by date. Space requirements are low, but consumption integrated with the original data becomes unwieldly.

Solution 2B) Using an in-house program, track when the diffs happen and store them in a sparsely filled table that looks much like the original table, filled only with the data that’s changed and the date when changed. Cons, model is sparse and consumption integrated with the original data is non-trivial.

I guess, basically, how do I integrate the dimension of time into a relational database, keeping in mind both the viewing of the data and awareness of differences between time periods?

Does this relate to data warehousing at all?

Smells like… Slowly changing dimension?

Answer 1

I had a similar problem – big flat files imported to the database once per day. Most of the data is unchanging.

Add two extra columns to the table, starting_date and ending_date. The default value for ending_date should be sometime in the future.

To compare one file to the next, sort them both by the key columns, then read one row from each file.

• If the keys are equal: compare the rest of the columns to see if the data has changed. If the row data is equal, the row is already in the database and there’s nothing to do; if it’s different, update the existing row in the database with an ending_date of today and insert a new row with a starting_date of today. Read a new row from both files.
• If the key from the old file is smaller: the row was deleted. Update ending_date to today. Read a new row from the old file.
• If the key from the new file is smaller: a row was inserted. Insert the row into the database with a starting_date of today. Read a new row from the new file.

Repeat until you’ve read everything from both files.

Now to query for the rows that were valid at any date, just select with a where clause test_date between start_date and end_date.