Question

I have a process that reads a data feed line by line, parses and inserts the data into a MyISAM table. When it first start, it goes really quick, probably around 1000 records a second. As time progresses it gets slower and slower, right now we’re at about 1 rows every 180 seconds.

The general syntax of the function is:

function parse($file) {
  $handle = fopen($file, 'r');
  while (!feof($handle)) {
    $line = fgets($fileHandle, 1000);
    switch (substr($line, 0, 2)) { //gets record type
      case '01' :
        //parse the record
        //escapes some strings with mysql_real_escape_string()
        mysql_query('INSERT INTO table VALUES ($a, $b, $c...');
      case '02' :
        ...
    }
  }
}

The current file being parsed has a few million records. The server doesn’t appear to be losing memory space. Does anybody know what could be causing the process to slow down?

Answer 1

It probably has to do with having to write so frequently to your indexes. You are already using MyISAM which would be my first suggestion.

Some suggestions

1. Batch insert every 100 rows
2. Use INSERT DELAYED to allow MySQL to allow inserts to quickly finish queuing, and MySQL will insert the record when resources are more permitting.
3. Have your script create an SQL dump/delimited file which you can load using IN FILE importing functions, see http://dev.mysql.com/doc/refman/5.1/en/load-data.html which will be MUCH FASTER than batch inserts