Question

I have a csv file that has roughly 100K entries I need to process and insert into a data base.

Previously it was very slow because it makes an SQL call for every entry. I do this though because if I try to build 1 single query to do this I will run out of memory.

I migrated to a new server and now I get an error every time I run it:

SQL Error : 2006 MySQL server has gone away

I am not sure but think this is just happening because how inefficient my code is.

What can I do to make it perform better and not get the error?

Here is the code:

//empty table before saving new feed
$model->query('TRUNCATE TABLE diamonds');

$fp = fopen($this->file,'r');

while (!feof($fp)) 
{
    $diamond = fgetcsv($fp);

    //skip the first line
    if(!empty($firstline))
    {
        $firstline = true;
        continue;   
    }

    if(empty($diamond[17]))
    {
        //no price -- skip it
        continue;
    }

    $data = array(
        'seller'             => $diamond[0],                             
        'rapnet_seller_code' => $diamond[1],                         
        'shape'              => $diamond[2],
        'carat'              => $diamond[3],
        'color'              => $diamond[4],
        'fancy_color'        => $diamond[5],
        'fancy_intensity'    => $diamond[6],
        'clarity'            => empty($diamond[8]) ? 'I1' : $diamond[8],
        'cut'                => empty($diamond[9]) ? 'Fair' : $diamond[9],
        'stock_num'          => $diamond[16],
        'rapnet_price'       => $diamond[17],
        'rapnet_discount'    => empty($diamond[18]) ? 0 : $diamond[18],
        'cert'               => $diamond[14],
        'city'               => $diamond[26],
        'state'              => $diamond[27],
        'cert_image'         => $diamond[30],
        'rapnet_lot'         => $diamond[31]
    );

    $measurements = $diamond[13];
    $measurements = strtolower($measurements);
    $measurements = str_replace('x','-',$measurements);
    $mm = explode('-',$measurements);

    $data['mm_width'] = empty($mm[0]) ? 0 : $mm[0];
    $data['mm_length'] = empty($mm[1]) ? 0 : $mm[1];
    $data['mm_depth'] = empty($mm[2]) ? 0 : $mm[2];

    //create a new entry and save the data to it.
    $model->create();
    $model->save($data);

}
fclose($fp);

Answer 1

You’re probably exceeding MySQL’s max_allowed_packet setting, which sets a hard limit (in bytes) on how long a query string can be. There’s nothing wrong with doing multi-value inserts, but 100k of them is definitely pushing things.

Instead of doing all 100k at once, try doing 1000 in a loop. You’re still reducing total query count (down from 100k to just 1000), so it’s still a net gain.