Question

A previous DBA managed a non relational table with 2.4M entries, all with unique ID’s. However, there are duplicate records with different data in each record for example:

+---------+---------+--------------+----------------------+-------------+
| id      | Name    | Address      | Phone   | Email      | LastVisited |
+---------+---------+--------------+---------+------------+-------------+
| 1       | bob     | 12 Some Road | 02456   |            |             | 
| 2       | bobby   |              | 02456   | bob@domain |             |
| 3       | bob     | 12 Some Rd   | 02456   |            | 2010-07-13  | 
| 4       | sir bob |              | 02456   |            |             |
| 5       | bob     | 12SomeRoad   | 02456   |            |             |
| 6       | mr bob  |              | 02456   |            |             |
| 7       | robert  |              | 02456   |            |             |
+---------+---------+--------------+---------+------------+-------------+

This isnt the exact table – the real table has 32 columns – this is just to illustrate

I know how to identify the duplicates, in this case i’m using the phone number. I’ve extracted the duplicates into a seperate table – there’s 730k entires in total.

What would be the most efficient way of merging these records (and flagging the un-needed records for deletion)?

I’ve looked at using UPDATE with INNER JOIN’s, but there are several WHERE clauses needed, because i want to update the first record with data from subsequent records, where that subsequent record has additional data the former record does not.

I’ve looked at third party software such as Fuzzy Dups, but i’d like a pure MySQL option if possible

The end goal then is that i’d be left with something like:

+---------+---------+--------------+----------------------+-------------+
| id      | Name    | Address      | Phone   | Email      | LastVisited |
+---------+---------+--------------+---------+------------+-------------+
| 1       | bob     | 12 Some Road | 02456   | bob@domain | 2010-07-13  | 
+---------+---------+--------------+---------+------------+-------------+

Should i be looking at looping in a stored procedure / function or is there some real easy thing i’ve missed?

Answer 1

U have to create a PROCEDURE, but before that
create ur own temp_table like :

Insert into temp_table(column1, column2,....) values (select column1, column2... from myTable GROUP BY phoneNumber) 

U have to create the above mentioned physical table so that u can run a cursor on it.

create PROCEDURE myPROC
{

create a cursor on temp::
fetch the phoneNumber and id of the current row from the temp_table to the local variable(L_id, L_phoneNum).

And here too u need to create a new similar_tempTable which will contain the values as

Insert into similar_tempTable(column1, column2,....) values (Select column1, column2,.... from myTable where phoneNumber=L_phoneNumber)

The next step is to extract the values of each column u want from similar_tempTable and update into the the row of myTable where id=L_id and delete the rest duplicate rows from myTable.

And one more thing, truncate the similar_tempTable after every iteration of the cursor…

Hope this will help u…