Home/ Questions/Q 939181
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-15T21:44:59+00:00

I have a couple of databases containing simple data which needs to be imported

  • 0

I have a couple of databases containing simple data which needs to be imported into a new format schema. I’ve come up with a flexible schema, but it relies on the critical data of the to older DBs to be stored in one table. This table has only a primary key, a foreign key (both int’s), a datetime and a decimal field, but adding the count of rows from the two older DBs indicates that the total row count for this new table would be about 200,000,000 rows.

How do I go about dealing with this amount of data? It is data stretching back about 10 years and does need to be available. Fortunately, we don’t need to pull out even 1% of it when making queries in the future, but it does all need to be accessible.

I’ve got ideas based around having multiple tables for year, supplier (of the source data) etc – or even having one database for each year, with the most recent 2 years in one DB (which would also contain the stored procs for managing all this.)

Any and all help, ideas, suggestions very, deeply, much appreciated,

Matt.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Most importantly. consider profiling your queries and measuring where your actual bottlenecks are (try identifying the missing indexes), you might see that you can store everything in a single table, or that buying a few extra hard disks will be enough to get sufficient performance.

    Now, for suggestions, have you considered partitioning? You could create partitions per time range, or one partition with the 1% commonly accessed and another with the 99% of the data.

    This is roughly equivalent to splitting the tables manually by year or supplier or whatnot, but internally handled by the server.

    On the other hand, it might make more sense to actually splitting the tables in ‘current’ and ‘historical’.

    Another possible size improvement is using an int (like an epoch) instead of a datetime and provide functions to convert from datetime to int, thus having queries like

    SELECT * FROM megaTable WHERE datetime > dateTimeToEpoch('2010-01-23')

    This size savings will probably have a cost performance wise if you need to do complex datetime queries. Although on cubes there is the standard technique of storing, instead of an epoch, an int in YYYYMMDD format.

    • 0