Question

I have a web app that reads data from a SQL DB that contains Message Tracking logs, for performance reasons I want to ensure there is never more than 2 days worth of logs in the DB.

Normally this is a simple challenge, just delete the records right with a delete statement?

Unfortunately not that simple for me, the table contains record counts in the millions and the delete statements take a while to run. Is there a way to make this any quicker?

I’m currently doing this:

DECLARE @CutOffDate datetime;
SET @CutOffDate = DATEADD(d, -2, GETDATE());

WHILE @@ROWCOUNT > 0
BEGIN
    DELETE TOP (2000) E12_MessageTracking
    WHERE [TimeStamp] < @CutOffDate;

    DELETE TOP (2000) E12_MessageTracking_Recipients
    WHERE [TimeStamp] < @CutOffDate;
END

I am running the bulk insert queries quite frequently so I need to be able to run the deletes frequently as well, but currently the task is “overrunning” on the delete statements and the scripts that run are not able to keep up with real time.

UPDATE:

This is the version info that you guys asked for:

Microsoft SQL Server 2008 R2 (RTM) – 10.50.1810.0 (X64) Feb 3 2012 17:19:10 Copyright (c) Microsoft Corporation Standard Edition (64-bit) on Windows NT 6.1 (Build 7600: ) (Hypervisor)

For the indexes, the tables don’t have a PK/FK relationship, but they “link” on a field called MessageID. The problem is MessageID is not always filled in so I am using a sub field called InternalMessageID which links better. One of the problems i’ve encountered is that there are multiple lines in the E12_MessageTracking table for each message, that link to the E12_MessageTracking_Recipients table.

I’d love to have a proper relationship between these tables but right now I can’t as I have no way of identifying all of the items that relate to a message without using the pre-existing InternalMessageID or MessageID columns.

For indexes, they are as follows:

E12_MessageTracking

Index on the InternalMessageID column, with included columns set for Timestamp and Event

E12_MessageTracking_Recipients

Index on the InternalMessageID column, with included columns set for Timestamp and Recipient

Thanks,
Chris.

Answer 1

First thing I would consider is to make sure the tables have an index where TimeStamp is the first indexed column.

More detail:

If the Timestamp isn’t the first column and your query doesn’t filter on the columns that come before it, then the index can’t be used (imagine looking for a book in a library where you know the first letter of the second word).

You can have a better idea by looking at the execution plan for the script in SSMS. If the query on E12_MessageTracking says a Table Scan, then no indexes are being used.