Question

MySQL/ASP – Delete Duplicate Rows
I have a table with 100,000 rows called ‘photoSearch’. When transferring the data from other tables (that took bloody ages and I was bloody tired), I accidentally forgot to remove the test transfer I did, which left 3500 rows in the table before I transferred everything over in one go.

The ID column is ‘photoID’ (INT) and I need to remove all duplicates that have a photoID of less than 6849. If I could just remove the duplicates, it would be less painful than to delete the table and start another transfer.

Has anybody got any suggestions on the most practical and safest way to do this?

---

UPDATE:

I actually answered my own question. I backed up my table for safety, and then I ran this:

ALTER IGNORE TABLE photoSearch ADD UNIQUE INDEX unique_id_index (photoID);

This removed all 3500 duplicates in under a minute 🙂

---

Answer 1

Traditional method

Backup your existing table photoSearch to something like tmp_photoSearch using a

create table tmp_photoSearch select * from photoSearch;

After that, you can perform data massage into table tmp_photoSearch.
Once you have gotten the results as expected,
perform a swap table

rename table photoSearch to photoSearch_backup, tmp_photoSearch to photoSearch;

---

To increase insert speed (if the bottle-neck is not on network transfer),

http://dev.mysql.com/doc/refman/5.0/en/insert-speed.html

To increase performance for MyISAM tables, for both LOAD DATA INFILE and INSERT, enlarge the key cache by increasing the key_buffer_size system variable