Question

I have a query that seems to take far too long to execute. It’s been a while (years) since I’ve done anything much beyond very simple select/update and v.simple joins so I’m more than a little rusty here!

SELECT count(distinct r.TAGCODE)
FROM RAWREADS r
where r.TAGCODE NOT IN (
    select distinct r.TAGCODE 
    from RAWREADS r, checkpoints c, guards g, INCIDENTITEMS i
        where r.TAGCODE = c.TAGNO
        or    r.TAGCODE = g.IDTAG
        or    r.TAGCODE = i.IDTAG
    );

The inner select seems to work correctly, if slowly (several seconds) but as soon as I add the outer ‘count where not in’ I end up having to kill my DB connection – so no good for working into an application! 😉

I hope the query above is clear what I’m trying to achieve… fetch all rawread tags where that tag does not match the respective columns in checkpoints/guards/incidentitems.

I’m using the Flamebird database server (no choice on that) and FlameRobin to run the query if that matters.

At some point I also need to add a query to the outer select to ensure I don’t select any rawreads where the tagcode is null or “”.

I have removed a “char_length(tagcode) > 0” criteria from the out select in the hope of speeding things along but I think my problem is more fundamental than that.

Answer 1

Many databases (and maybe Firebird as well) cannot easily optimize NOT IN conditions. So you might try to rerwrite this as a NOT EXTISTS.

Additionally looking at the inner select, I think the way you are joining there is producing too many rows. It’s not a real join but not a cartesian product either. Anyway I could imagine that doing a UNION in there is also more efficient.

So try this:

SELECT count(distinct r.TAGCODE)
FROM RAWREADS r
where NOT EXISTS (
    select 1
    from checkpoints c
    where c.tagno = r.tagcode
    union all
    select 1 
    from guards g
    where g.idtag = r.tagcode
    union all 
    select 1 
    from INCIDENTITEMS i
    where i.idtag = r.tagcode);

If you have indexes on checkpoints(tagno), guards(idtag) and incidentitems(idtag) that should run fairly quick.