Question

I have this query that’s running too slow. I’m not sure what all info I should provide in order to make it easy for you to help me, but I’ll take a stab at it and then add more when you big-brains inevitably ask for stuff that I either didn’t think to include or don’t know what is.

I want to identify customers (but using only part of their address — accommodating households and businesses) who first registered a purchase in 2006.

My first attempt was:

select
    distinct a.line1 + '|' + substring(a.zip,1,5)
from 
    registrations r
    join customers c on r.custID = c.id
    join addresses a on c.addressID = a.id
where year(r.purchaseDate) = 2006
    and a.line1 + '|' + substring(a.zip,1,5) not in (
        select
            distinct a.line1 + '|' + substring(a.zip,1,5)
        from
            registrations r
            join customers c on r.custID = c.id
            join addresses a on c.addressID = a.id
        where
            year(r.purchaseDate) < 2006
    )

and when it was running too long, I switched a NOT EXISTS (with which I am less comfortable, but willing to experiment) like

select
    distinct a.line1 + '|' + substring(a.zip,1,5)
from
    registrations r
    join customers c on r.custID = c.id
    join addresses a on c.addressID = a.id
where
    year(r.purchaseDate) = 2006
    and not exists (
        select
            1
        from
            registrations r
            join customers c on r.custID = c.id
            join addresses ia on c.addressID = ia.id
        where
            ia.line1 + '|' + substring(ia.zip,1,5) = a.line1 + '|' + substring(a.zip,1,5) and
            year(r.purchaseDate) < 2006
        )
group by
    a.line1 + '|' + substring(a.zip,1,5)

But it runs too long too. Like no results in 17 hours kind of too long. I think the first thing to consider is where my SQL might be wrong or sub-optimal but in case it’s not that, I want to also give you enough info to consider the environment.

So, diagnostic info. You probably don’t care, but just in case: it’s running on a G6 server with four quads and 20 GB RAM; each query is limited to occupying four processors to keep performance up for requests from the web server; when I’m running this query, we’re clearing off other big imports and reports because of deadlocks but the web server is customer-facing and can’t be stopped.) There are roughly: 15 million registrations, 11 million customers and 8.6 million addresses. I rebuilt all of the indexes just to be sure that fragmentation wasn’t the problem. However, I’m not really sure how to index properly, so I’m totally open to this being a problem — some of these indexes were a result of me futzing around and some were scripts that one of the MS analysis tools gave me to improve performance. I’m also not exactly sure how to convey index info to you, so I’ll just give the create scripts:

ALTER TABLE [dbo].[registrations] ADD  CONSTRAINT [PK_flatRegistrations_1] PRIMARY KEY CLUSTERED 
(
    [Id] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]

.

ALTER TABLE [dbo].[customers] ADD  CONSTRAINT [PK_flatCustomers_1] PRIMARY KEY CLUSTERED 
(
    [Id] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]

.

ALTER TABLE [dbo].[addresses] ADD  CONSTRAINT [PK_addresses] PRIMARY KEY CLUSTERED 
(
    [ID] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]


CREATE NONCLUSTERED INDEX [addresses] ON [dbo].[addresses] 
(
    [line1] ASC,
    [line2] ASC,
    [city] ASC,
    [state] ASC,
    [zip] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]


CREATE NONCLUSTERED INDEX [deliverable] ON [dbo].[addresses] 
(
    [addressDeliverable] ASC
)WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]


CREATE NONCLUSTERED INDEX [_dta_index_addresses_5_1543676547__K9_K1_6] ON [dbo].[addresses] 
(
    [addressDeliverable] ASC,
    [ID] ASC
)
INCLUDE ( [zip]) WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]


CREATE NONCLUSTERED INDEX [_dta_index_addresses_5_1543676547__K1_K9_6] ON [dbo].[addresses] 
(
    [ID] ASC,
    [addressDeliverable] ASC
)
INCLUDE ( [zip]) WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]


CREATE NONCLUSTERED INDEX [_dta_index_addresses_5_1543676547__K1_6] ON [dbo].[addresses] 
(
    [ID] ASC
)
INCLUDE ( [zip]) WITH (PAD_INDEX  = OFF, STATISTICS_NORECOMPUTE  = OFF, SORT_IN_TEMPDB = OFF, IGNORE_DUP_KEY = OFF, DROP_EXISTING = OFF, ONLINE = OFF, ALLOW_ROW_LOCKS  = ON, ALLOW_PAGE_LOCKS  = ON) ON [PRIMARY]

Thanks a bunch for your time!

Answer 1

I think your Not Exists subquery’s table alias is wrong. Try this:

select  r.custID,
        a.line1 + '|' + substring(a.zip,1,5) 
from    registrations r     
join    customers c on r.custID = c.id     
join    addresses a on c.addressID = a.id 
where   r.purchaseDate between '2006-01-01' and '2006-12-31'      
and not exists (         
        select  1         
        from   registrations ir             
        join customers ic on ir.custID = ic.id           
        join addresses ia on ic.addressID = ia.id         
        where   ia.line1 = a.line1
        and     substring(ia.zip,1,5) = substring(a.zip,1,5) 
        and     ir.purchaseDate < '2006-12-31'        
        )