Question

The current DB or our project has crossed over 40 GB this month and on an average it is growing monthly by around 3 GB. Now all the tables are best normalized and proper indexing has been used. But still as the size is growing it is taking more time to fire even basic queries like ‘select count(1) from table’. So can u share some more points that will help in this front. Database is Sql Server 2005. Further if we implement Partitioning wouldn’t it create a overhead ?

Thanks in advance.

Answer 1

1. make sure you have suitable/appropriate indexes
2. make sure you have a good index maintenance strategy (e.g. rebuild/defrag/keep statistics up to date to ensure indexes stay performing well)
3. identify poorly performing queries and optimise them (may have been written/tested against small data volumes when performance issues would not have shown up)
4. consider partitioning your data (e.g. SQL 2005 and onwards has built in support for partitioning if you have Enterprise Edition). Edit: to elaborate on SQL Server partitioning, I full recommend a read through this MSDN article on the whys and the hows. On a general note, there was also a good talk at QCon 2008 by Randy Shoup (eBay architect) on scalability, of which one of the key points on scaling a system in general is to partition. It’s summarised here.
5. is your db server hardware sufficient? could it benefit from more memory?
   Edit: looking at your comment with your hardware info, I think you could do with (at least) throwing more RAM in it
6. you may benefit from some denormalisation. Difficult to be specific without knowing exact db structure, but denormalising may improve certain queries at the expense of data duplication/disk space