Question

We’ve got a database table that need a multiple column unique key. However, one of those columns is TEXT and has lengths as long as 1000 chars (so varchar won’t work either). Because of the TEXT column, I can’t actually have a unique key for those columns. What’s a good way to remove duplicates? Of course, fast would be nice.

Answer 1

The best way is to use a UNIQUE INDEX to avoid duplicate.

Creating a new unique key on the over columns you need to have as uniques will automatically clean the table of any duplicates.

ALTER IGNORE TABLE `table_name`
    ADD UNIQUE KEY `key_name`(`column_1`,`column_2`);

The IGNORE part does not allow the script to terminate after the first error occurs. And the default behavior is to delete the duplicates.