Question

I have a thousands of data parsed from huge XML to be inserted into database table using PHP and MySQL. My Problem is it takes too long to insert all the data into table. Is there a way that my data are split into smaller group so that the process of insertion is by group? How can set up a script that will process the data by 100 for example? Here’s my code:

foreach($itemList as $key => $item){
     $download_records  = new DownloadRecords();
    //check first if the content exists
    if(!$download_records->selectRecordsFromCondition("WHERE Guid=".$guid."")){
         /* do an insert here */
    } else {
         /*do an update */
    }

}

*note: $itemList is around 62,000 and still growing.

Answer 1

Using a for loop?

But the quickest option to load data into MySQL is to use the LOAD DATA INFILE command, you can create the file to load via PHP and then feed it to MySQL via a different process (or as a final step in the original process).

If you cannot use a file, use the following syntax:

insert into table(col1, col2) VALUES (val1,val2), (val3,val4), (val5, val6)

so you reduce to total amount of sentences to run.

EDIT: Given your snippet, it seems you can benefit from the INSERT … ON DUPLICATE KEY UPDATE syntax of MySQL, letting the database do the work and reducing the amount of queries. This assumes your table has a primary key or unique index.

To hit the DB every 100 rows you can do something like (PLEASE REVIEW IT AND FIX IT TO YOUR ENVIRONMENT)

$insertOrUpdateStatement1 = "INSERT INTO table (col1, col2) VALUES ";
$insertOrUpdateStatement2 = "ON DUPLICATE KEY UPDATE ";
$counter = 0;
$queries = array();

foreach($itemList as $key => $item){
    $val1 = escape($item->col1); //escape is a function that will make 
                                 //the input safe from SQL injection. 
                                 //Depends on how are you accessing the DB

    $val2 = escape($item->col2);

    $queries[] = $insertOrUpdateStatement1. 
    "('$val1','$val2')".$insertOrUpdateStatement2.
    "col1 = '$val1', col2 = '$val2'";

    $counter++;

    if ($counter % 100 == 0) {
        executeQueries($queries);
        $queries = array();
        $counter = 0;
    }
}

And executeQueries would grab the array and send a single multiple query:

function executeQueries($queries) {
   $data = "";
     foreach ($queries as $query) {
        $data.=$query.";\n";
    }
    executeQuery($data);
}