Question

I have a PHP script to import various data from text files.

The import is very complex and my test file has 32.000 entrys. These entrys have to be parsed and inserted into a mysql database.

If i will run my script it needs 30 minutes to finish… And at this time my server cpu is 99% idle.

Is there a chance to optimize php and mysql that they are using more power from the machine?

code:

if ($handle = @fopen($filename, "r")) {

    $canceled = false;
    $item = null;
    $result = null;

    while (!feof($handle)) {
        $buffer = fgets($handle, 4096);
        $line = $buffer;

        if (substr($line, 0, 2) == '00') {

            continue;
        }
        else if (substr($line, 0, 2) == '99') {
            continue;
        }
        else if (strlen($line) < 75) {
            continue;
        }

        Reread:
        if ($canceled) {
            break;
        }

        if ($item == null) {
            $item = new DA11_Item();
            $item->boq_id = $boq_id;
        }
        $result = $this->add_line_into_item($item, $line);

        if ($result == self::RESULT_CLOSED) {
            $this->add_item($item);
            $item = null;
        }
        else if ($result == self::RESULT_REREAD) {
            $this->add_item($item);
            $item = null;
            goto Reread;
        }
        else if ($result == self::RESULT_IGNORD) {
            if (count($item->details()) > 0) {
                    $this->add_item($item);
            }
            $item = null;
        }
    }

    if ($item !== NULL) {
        $this->add_item($item);
    }

    fclose($handle);
}

add_item will perform a $item->save() and saves it to the database.

thx and kind regards,
viperneo

Answer 1

One problem you have is that every single insert is a separate request to your db-server including it’s response. With 32.000 records you maybe get an idea, that this is a quite huge overhead. Use bulk inserts for (lets say) 1000 records at once

INSERT INTO foo (col1, col2) VALUES
  (1,'2'),
  (3,'4')
  -- 997 additional rows
  (1999, '2000');

Additional Transactions may help

Update, because you mentioned active-record: I recommend to avoid any additional abstraction layer for such mass import tasks.