Question

I know this might be redundant but I have had the same query running for almost 3 days and before I kill it, I would like to get a community sanity check.

DELETE
FROM    mytble
WHERE   ogc_fid NOT IN
    (SELECT     MAX(dup.ogc_fid)
        FROM        mytble As dup
        GROUP BY    dup.id)

mytble is the name of the table, ogc_fid is the name of the unique id field and id is the name of the field that I want to be the unique id. There are 41 million records in the table and indexes are built and everything so I am still a bit concerned about why its taking so long to complete. Any thoughts on this?

Answer 1

If I understood correctly, you want to delete all the records for which a record with the same dup_id
(but with a higher ogc_fid) exists. And keep only those with the highest ogc_fid.

-- DELETE -- uncomment this line and comment the next line if proven innocent.
SELECT COUNT(*)
  FROM   mytble mt
 WHERE   EXISTS (
  SELECT *
    FROM mytble nx
   WHERE nx.dup_id = mt.dup_id    -- there exists a row with the same dup_id
     AND nx.ogc_fid > mt.ogc_fid  -- , ... but with a higher ogc_fid 
);

With an index on dup_id (and maybe on ogc_id) this should run maybe a few minutes for 41M records.

UPDATE: if no indexes exist, you could speed up the above queries by first creating an index:

 CREATE UNIQUE INDEX sinterklaas ON mytble (dup_id, ogc_id);