Question

We have a giant SQL Server 2005 database (75GB) which basically is just data in one table with sales values (per day, store and article). We want to make place by adding up the sales value per week for every record older than a year (still grouped per store and article). So in theory for data older than a year we can then remove 6 out 7 records.

Writing a procedure to do this is not really a problem but it runs like forever. So I was looking for a strategy that could make this run in a reasonable amount of time.

To give you an idea: running SELECT count(*) ran for over 4 minutes

We do have a few indexes (on date (clustered) and on the store, article and date combination). Adding any more indexes also takes like forever.

Anyone has got a good strategy on how to perform this task? Any suggestions on TSQL methods which perform better than the basic DML statements?

Answer 1

If You use SQL Server 2005 Enterprise Edition, You should consider using partitioning feature. Benefits:

• by partitioning your data on date column, queries will run much faster because SQL Server will access only specific partition; this way you can run your day->week procedure on date range and it will run much faster (and run multiple procedures on different date ranges at the same time)
• if You want to keep your daily data, just move old partitions to slower storage (hard disk)
• your procedure should prepare weekly data in new table and then switch partitions – it is much faster than delete daily data and insert weekly data

If You don’t use Enterprise Edition, use this link to see partitioning (sharding or horizontal partitioning) capabilities not based on SQL Server 2005 partitioning feature.

For stored procedure optimization:

• reevaluate current indexes for your SP
• consider daily->week procedure to run on date ranges, for instance, year by year or month by month – running procedure on whole history will be a lot of work for SQL Server and underlying hardware
• probably the best way is: following previous item about date ranges, create new table based on old weekly data and recent daily data then create indexes and then in one transaction drop original table and use sp_rename to put old table instead of new one – rename is almost instant so no one will notice the delay if that is important
• consider dropping indexes on target table because inserts will be much slower with it – only if You are working on the original table (delete + insert)

Off-topic hint: if using Enterprise Edition, consider compressing your table since SQL Server 2005 is usually good at compression of fact tables – you will probably gain both performance and disk space if You have enough CPU power.