Home/ Questions/Q 7562539
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-30T13:25:57+00:00

I have a MySql database with just over 30 Million rows, but some of

  • 0

I have a MySql database with just over 30 Million rows, but some of the rows (appropriately 3 million) are duplicate i am using the php script below to copy the database and check for duplicates, but its is running very slow its taken 3 days and its not even half way throught, is there anyway to speed it up? i’m running the script on the MySql Server itself.

    <?php

$dbhost = "localhost" ;


$con = mysql_connect("$dbhost","$dbuser","$dbpass");
if (!$con)
  {
  die('Could not connect: ' . mysql_error());
  }

mysql_select_db("$dbame", $con);

$server_self = $_SERVER['PHP_SELF'];
$rows_per_page = 1000;
$dbname_var = post1;




//paging script
$query = "SELECT count(*) FROM $dbname_var";         //change!!
//paging script



//PAGING SCRIPT
if (isset($_GET['pageno'])) {
   $pageno = $_GET['pageno'];
} else {
   $pageno = 1;
} // if


$resulta = mysql_query($query) or trigger_error("SQL", E_USER_ERROR);
$query_data = mysql_fetch_row($resulta);



$numrows = $query_data[0];
$lastpage      = ceil($numrows/$rows_per_page);

$pageno = (int)$pageno;
if ($pageno > $lastpage) {
   $pageno = $lastpage;
} // if
if ($pageno < 1) {
   $pageno = 1;
} // if

$limit = 'LIMIT ' .($pageno - 1) * $rows_per_page .',' .$rows_per_page;
$nextpage = $pageno+1;
$prevpage = $pageno-1;







$result = mysql_query("SELECT * FROM $dbname_var $limit");
while($row = mysql_fetch_array( $result )) {


$as = $row[1];
$ad = $row[2];
$af = $row[10];

//CHECK IF THE ROW ALREADY EXISTS IN THE TABLE
$resultdb1 = mysql_query("SELECT * FROM post_final WHERE
add1='$as' AND
add2='$ad' AND
add10='$af'");

$num_rowsdb1 = mysql_num_rows($resultdb1);

//IF IT DOSENT ADD IT!
if($num_rowsdb1 < 1) {

mysql_query("INSERT INTO post_final (add1,
 add2,
 add3,
 add4,
 add5,
 add6,
 add7,
 add8,
 add9,
 add10)
VALUES ('$row[1]',
 '$row[2]',
 '$row[3]',
 '$row[4]',
 '$row[5]',
 '$row[6]',
 '$row[7]',
 '$row[8]',
 '$row[9]',
 '$row[10]')");
} }







$nxt = $_GET['pageno'] + 1;



if ($_GET['pageno'] < $lastpage) {

print "<BODY>
<script type=\"text/javascript\">
<!--
window.location = \"/addnew.php?pageno=$nxt&lastpage=$lastpage\"
//-->
</script>
</BODY>
</HTML>\n";  }

?>

#   Column  Type    Collation   Attributes  Null    Default Extra   Action
 1  ID  int(10)         No  None    AUTO_INCREMENT    Change      Drop   More 
 2  add1    varchar(50) latin1_german2_ci       No  None          Change      Drop   More 
 3  add2    varchar(50) latin1_german2_ci       No  None          Change      Drop   More 
 4  add3    varchar(50) latin1_german2_ci       No  None          Change      Drop   More 
 5  add4    varchar(50) latin1_german2_ci       No  None          Change      Drop   More 
 6  add5    varchar(50) latin1_german2_ci       No  None          Change      Drop   More 
 7  add6    varchar(50) latin1_german2_ci       No  None          Change      Drop   More 
 8  add7    varchar(50) latin1_german2_ci       No  None          Change      Drop   More 
 9  add8    varchar(50) latin1_german2_ci       No  None          Change      Drop   More 
 10 add9    varchar(50) latin1_german2_ci       No  None          Change      Drop   More 
 11 add10   varchar(50) latin1_german2_ci       No  None          Change      Drop   More 

Action  Keyname Type    Unique  Packed  Column  Cardinality Collation   Null    Comment
 Edit    Drop   PRIMARY BTREE   Yes No  ID  10044279    A       
 Edit    Drop   add10   BTREE   No  No  add10   590839  A       
 Edit    Drop   add1    BTREE   No  No  add1    264323  A       
 Edit    Drop   add2    BTREE   No  No  add2    233587  A
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Use only SQL, there’s no need to take the data back and forth between SQL and PHP. Assuming you have a database db1 with table post_final, create database db2 create empty post_final table and do

    INSERT INTO `db2`.`post_final` (add1, add2, ...., add10) 
SELECT DISTINCT add1, add2, ... , add10 FROM `db1`.`post_final`
    • 0