Question

I have a database table with fields like:

username, description, password.

Sometimes, members copy the description from one-another, to save time

So I have this:

John – John’s description – John’s password

Michael – John’s description – Michael’s password

Is there a mysql query that searches for duplicate field entries and deletes them?
How about deleting the entire row of data while we’re at it?

Answer 1

Create a new table, move non-duplicate entries in it, remove the old table and rename a new one.

Example:

CREATE TABLE `new_table` as
    SELECT * FROM `old_table` WHERE 1 GROUP BY [columns to remove duplicates by];

DROP TABLE `old_table`;

RENAME TABLE `new_table` TO `old_table`;

However, it’s good for periodically uses only, and it doesn’t check is there are any duplicates. It just groups unique entries and moves them to another table. It’s useful when you want to filter your entries.

---

Another way is to check like this:

SELECT `name` FROM `table` WHERE `description` LIKE '%descriptiontexthere%'

Then, if some results are found, it’s a duplicate.
However, there’s a big disadvantage: user can change only 1 letter and query will fail.
That disadvantage, however, can be avoided (not fully) by splitting a description into an array. For example, split on every 100 characters and then check like in above example, but with multiple conditions (e.g. description LIKE 'first100chars' OR description LIKE 'second100chars') .

---

The third way is to split a description to an array of words and then select rows with too many same words. Row with X same words could be a possible duplicate. You can set the treshold based on length of an entered description.

You can never be sure if it’s duplicate or not, unless it’s completely the same entry.