Question

I have the following code (more or less) to import anywhere from 500.000 to 4.000.000 rows:

$sSql = "Insert into table (a,b,c) VALUES(?,?,?)"
$oSQLStmnt = $pdo->prepare($sSql);
$oSQLStmnt->setAttribute(PDO::SQLSRV_ATTR_ENCODING, PDO::SQLSRV_ENCODING_SYSTEM);
if (!$oSQLStmnt) {
    echo $pdo->errorInfo(); // Handle errors
}
$pdo->beginTransaction();
$iLineCounter = 1;
while (($sLine = fgets ($oCSV, 8000)) !== FALSE) {
      $aLine = explode('|', $sLine); //Fgetscsv did not work properly 
       if ($iLineCounter % 100 == 0) {
            lo("Inserting row " . $iLineCounter);
            $pdo->commit();
            sleep(0.15);
            $pdo->beginTransaction();
       }
       try {
            $oSQLStmnt->execute($aLine);
            $iSuccesulInserts++;
       }
       catch (exception $e) {
            print_r($e);
            $iFailedInserts++;
       }

       $iLineCounter++;
}
$pdo->commit();

As you can see, I perform a commit every 100 lines, and I even added some sleep. I used to run the commit only once every 25.000 lines, and I did not use any sleep. However, at one point, I discovered I was missing records. I started playing with these settings (sleep and number of rows). This way I reduced the number of missing records from 50.000 to about a 100. But I’m still missing records! Where are they going? I know the SQL is ok, because I immediately receive errors when somethings wrong there.

I thought I could stack a lot of inserts during a transaction? Could calling beginTransaction be a problem?

UPDATE:

The bounty ended and I had to award it. Thank you all for your answers. Or tips actually, as none of you actually answered my question. I was not asking for a workaround, although you suggestions are much appreciated. The answer the bounty was awarded to received it because it came closest to actually answering my question. Unfortunately it did not work.

For now I’m using CSV bulk import, that works fine, but if anyone has any other tips for fixing this issue, please let me know. As I prefer using my original method.

Answer 1

Have you considered using Sprocs instead of insert statements? writing ANY number of records sequentially- one at a time- is kindof a waste of time / energy.. it’s just not as fast as it should be.

Are you sure you can’t use BULK INSERT or XML instead to insert multiple rows at a time?