Question

I am trying to read 738627 records from a flat file into MySQl. The script appears to run fine, but is giving me the above memory errors.

A sample of the file is:

#export_dategenre_idapplication_idis_primary
#primaryKey:genre_idapplication_id
#dbTypes:BIGINTINTEGERINTEGERBOOLEAN
#exportMode:FULL
127667880285760002817317350
127667880285760002818261461
127667880285760002825372301
127667880285760002827785570
127667880285760002827930241
127667880285760002827987861
127667880285760002828089791
127667880285760002828168361
127667880285760002828192041
127667880285760002829144541
127667880285760002829351511

I have tried increasing the allowed memory using

ini_set("memory_limit","80M");

and it still fails. Do I keep upping this until it runs?

The code in full is

  <?php
    ini_set("memory_limit","80M");
    $db = mysql_connect("localhost", "uname", "pword");
    // test connection 
    if (!$db) { 
        echo "Couldn't make a connection!"; 
        exit; 
    } 
    // select database 
    if (!mysql_select_db("dbname",$db))

    {
        echo "Couldn't select database!"; 
        exit; 
    }
    mysql_set_charset('utf8',$db);

    $delimiter = chr(1);
    $eoldelimiter = chr(2) . "\n";
    $fp = fopen('genre_application','r');
    if (!$fp) {echo 'ERROR: Unable to open file.</table></body></html>'; exit;}

 $loop = 0;
while (!feof($fp)) {
  $loop++;
    $line = stream_get_line($fp,128,$eoldelimiter); //use 2048 if very long lines
if ($line[0] === '#') continue;  //Skip lines that start with # 
    $field[$loop] = explode ($delimiter, $line);
    $fp++;
$export_date = $field[$loop][0];
$genre_id = $field[$loop][1];
$application_id = $field[$loop][2];

$query = "REPLACE into genre_apps
(export_date, genre_id, application_id)
VALUES ('$export_date','$genre_id','$application_id')";
print "SQL-Query: ".$query."<br>";
       if(mysql_query($query,$db))
         {
          echo " OK !\n";
         }
        else
             {
              echo "Error<br><br>";
              echo mysql_errno() . ":" . mysql_error() . "</font></center><br>\n";
             }

    }

    fclose($fp);
    ?> 

Answer 1

Your loop fills the variable $field for no reason (it writes to a different cell on every loop iteration), thereby using up more memory with every line.

You can replace:

$field[$loop] = explode ($delimiter, $line);
$export_date = $field[$loop][0];
$genre_id = $field[$loop][1];
$application_id = $field[$loop][2];

With:

list($export_date, $genre_id, $application_id) = explode($delimiter, $line);

For improved performance, you could take advantage of the ability to insert several lines using REPLACE INTO by grouping N lines into a single query.