Question

How can I run a query that finds duplicates between rows? It needs to not match one field but multiple.

Here is the EXPLAIN of the table.

+-------------+--------------+------+-----+-------------------+----------------+
| Field       | Type         | Null | Key | Default           | Extra          |
+-------------+--------------+------+-----+-------------------+----------------+
| id          | int(11)      | NO   | PRI | NULL              | auto_increment | 
| token       | varchar(64)  | NO   | MUL | NULL              |                | 
| maxvar      | float        | NO   |     | NULL              |                | 
| maxvbr      | float        | NO   |     | NULL              |                | 
| minvcr      | float        | NO   |     | NULL              |                | 
| minvdr      | float        | NO   |     | NULL              |                | 
| atype       | int(11)      | NO   |     | NULL              |                | 
| avalue      | varchar(255) | NO   |     | NULL              |                | 
| createddate | timestamp    | NO   |     | CURRENT_TIMESTAMP |                | 
| timesrun    | int(11)      | NO   |     | NULL              |                | 
+-------------+--------------+------+-----+-------------------+----------------+

I need to match all rows that match: token,maxvar,maxvbr,minvcr,minvdr,type and avalue. If all of those fields match those in another row then treat it as a “duplicate”.

Ultimately I want to run this as a delete command but I can easily alter the select.

UPDATE Still looking for solution that deletes with single query in MySQL

Answer 1

This query will find all duplicate records which should be deleted –

SELECT t1.id FROM table_duplicates t1
  INNER JOIN (
    SELECT MIN(id) id, token, maxvar, maxvbr, minvcr, minvdr, atype, avalue FROM table_duplicates
    GROUP BY token, maxvar, maxvbr, minvcr, minvdr, atype, avalue
    HAVING COUNT(*) > 1
  ) t2
  ON t1.id <> t2.id AND t1.token = t2.token AND t1.maxvar=t2.maxvar AND t1.maxvbr = t2.maxvbr AND t1.minvcr = t2.minvcr AND t1.minvdr = t2.minvdr AND t1.atype = t2.atype AND t1.avalue = t2.avalue;

This query will remove all duplicates –

DELETE t1 FROM table_duplicates t1
  INNER JOIN (
    SELECT MIN(id) id, token, maxvar, maxvbr, minvcr, minvdr, atype, avalue FROM table_duplicates
    GROUP BY token, maxvar, maxvbr, minvcr, minvdr, atype, avalue
    HAVING COUNT(*) > 1
  ) t2
  ON t1.id <> t2.id AND t1.token = t2.token AND t1.maxvar=t2.maxvar AND t1.maxvbr = t2.maxvbr AND t1.minvcr = t2.minvcr AND t1.minvdr = t2.minvdr AND t1.atype = t2.atype AND t1.avalue = t2.avalue;