Question

We have a database called AVL in SQL Server 2008 R2 SE. This database has many tables, but there is one in particular called ASSETLOCATION that has 46 millon rows right now and accounts for 99.9% of the total database size.

This table has information from 2008 to date, and the actual rate of growing is about 120k records daily.
Now, there are 2 situations we like to address:

• Performance is starting to degrade slowly, and everything is
  optimized so there’s not much to do
• Backup times are increasing and is becoming a problem (we do 1 full
  backup everyday). The BAK file is 11GB, after winrar does his thing
  the final size 2GB and then a script sends the file offsite. We have
  a T1 and pulling 2GB through the wire is taking about 5 hours.

All this is normal but here’s the catch I want to capitalize on: +90% of SQL statements use information only 3 months old or less, in other words, data from 2008, 2009 and 2010 don’t get accessed often.

I was thinking on creating one new database for each year. Lets say:
– AVL2008 database, only table there will be ASSETLOCATION with records from 2008
– AVL2009 database, only table there will be ASSETLOCATION with records from 2009
– AVL2010 database, only table there will be ASSETLOCATION with records from 2010

As you have already guessed data from the past don’t get changed, so this will be great from the backup perspective since the AVL database will only have the records from the current year. This approach will also help performance a lot.

Now for the problem. Assume the ASSETLOCATION table has the following columns:
– IDASSETLOCATION (int, PK identity)
– IDASSET (int, FK to ASSET table)
– WHEN (datetime)
– LATLONG (varchar(22), spatial info)

I need to create a view in the AVL database called “vASSETLOCATION”, witch is quite simple, but I don’t want the view accessing all the databases and joining the ASSETLOCATION tables via UNION, rather the only ones needed based on the WHEN field. For example:

select * from vASSETLOCATION where [WHEN] between '2008-01-01' and '2008-01-02'

In this case the view should ONLY ACCESS the AVL2008.ASSETLOCATION table

select * from vASSETLOCATION where [WHEN] between '2008-12-29' and '2009-01-05'

In this case the view should access AVL2008.ASSETLOCATION and AVL2009.ASSETLOCATION

select * from vASSETLOCATION where 
    ([WHEN] between '2008-01-01' and '2008-01-01')
or 
    ([WHEN] = getdate())

In this case the view should access AVL2008.ASSETLOCATION and AVL.ASSETLOCATION

I know a table scalar UDF in place of the view will solve the problem, but there are more than only 4 fields and [WHEN] is not the only field we may want to include in the where part.
Before anyone suggest it, the table partitioning feature will perhaps help in performance, but NOT in the backup problem.

If there a way to do this in a view?
Thanks.-

Answer 1

This sounds like a classic case for table partitioning or distributed partitioned views.

However you can work around this without ponying up the price for Enterprise edition (or doing all the prep work required to support those features) using some smart code that looks at the problem a little differently. You don’t want a single view that accesses all of the tables across the different databases, but what if you had multiple views and a stored procedure to control how they’re accessed?

Create views for the most common access patterns. Perhaps you have a view that covers date ranges for 2008-2010, 2008-2009, 2009-2010, etc. They might look like this:

CREATE VIEW dbo.vAL_2008_2009 
AS
    SELECT * FROM AVL2008.dbo.ASSETLOCATION
    UNION ALL
    SELECT * FROM AVL2009.dbo.ASSETLOCATION;
GO

CREATE VIEW dbo.vAL_2008_2010
AS
    SELECT * FROM AVL2008.dbo.ASSETLOCATION
    UNION ALL
    SELECT * FROM AVL2009.dbo.ASSETLOCATION
    UNION ALL
    SELECT * FROM AVL2010.dbo.ASSETLOCATION;
GO
-- etc. etc.

Now your code that handles the queries can take the input date parameters and calculate which view it needs to query. For example:

CREATE PROCEDURE dbo.DetermineViews
    @StartDate     DATETIME,
    @EndDate       DATETIME,
    @optionalToday BIT = 0
AS
BEGIN
    SET NOCOUNT ON;

    DECLARE @sql NVARCHAR(MAX) = N'';

    SET @sql = @sql + N'SELECT * FROM ' + CASE
      WHEN @StartDate >= '20080101' AND @EndDate < '20090101' THEN 'AVL2008.dbo.ASSETLOCATION'
      WHEN @StartDate >= '20080101' AND @EndDate < '20100101' THEN 'dbo.vAL_2008_2009'
      WHEN @StartDate >= '20080101' AND @EndDate < '20110101' THEN 'dbo.vAL_2008_2010'
      -- etc. etc.

      WHEN YEAR(@StartDate) = YEAR(CURRENT_TIMESTAMP) THEN 'AVL.dbo.ASSETLOCATION'
    ELSE '' END;

    IF @OptionalToday = 1 AND YEAR(@StartDate) <> YEAR(CURRENT_TIMESTAMP)
    BEGIN
        SET @sql = @sql + N'UNION ALL SELECT * FROM AVL.dbo.ASSETLOCATION'
    END

    SET @sql = @sql + ' WHERE [WHEN] BETWEEN ''' 
        + CONVERT(CHAR(8), @StartDate, 112) + ''' AND '''
        + CONVERT(CHAR(8), @EndDate,   112) + '''';

    IF @OptionalToday = 1
    BEGIN
        SET @sql = @sql + ' OR ([WHEN] >= DATEDIFF(DAY, 0, CURRENT_TIMESTAMP)
            AND [WHEN] < DATEADD(DAY, 1, DATEDIFF(DAY, 0, CURRENT_TIMESTAMP)';
    END

    PRINT @sql;
    -- EXEC sp_executeSQL @sql;
END
GO

I’m probably missing some of your business logic, and you’ll certainly want to add some error-handling in there and test the junk out of it, but this is a relatively easy-to-maintain solution that only requires updating that coincides with creating a new database to archive last year’s data, which it sounds like only happens once a year.