Question

I have a CSV file with around 1.8million rows. I need to insert them into a MySQL table from my PHP script. I am inserting the values in batches of 10,000.
The scripts run for very long time and crashes after inserting 80-95 batches. I have tried mysql_unbuffered_query() also but of no use.

if ($fp) {
        $batch = 1;
        $row_count = 1;
        $bucket_counter = 1;
        $mobile_numbers = array();
        $row_count_for_DB_write = 0;
        foreach ($campaign_numbers as $value) {
            $number = array($value);
            fputcsv($fp, $number);
            $row_count_for_DB_write++;
            $value_row = new stdClass();
            $value_row->number = $value;
            $value_row->bucket_number = $bucket_counter;
            $mobile_numbers[] = $value_row;
            if ($row_count == $bucket_size && $bucket_counter < $bucket_count) {
                $bucket_counter++;
                $row_count = 1;
                fclose($fp);
                $fp = fopen($directory . "/cn_$bucket_counter.csv", 'w');
                $logger->debug('Created csv file : ' . $directory . '/cn_$bucket_counter.csv');
            }

            if ($row_count_for_DB_write == CONSTANTS::BATCH_SIZE) {
                $logger->debug($batch." Batch insert starting at: ".date('d-m-Y_H-i-s', time()));
                $insert_count = $data_service->add_to_mobile_numbers_table($mobile_numbers_table, $mobile_numbers);
                $batch++;
                $logger->debug("Batch insert ending at: ".date('d-m-Y_H-i-s', time()));
                $row_count_for_DB_write = 1;
                unset($mobile_numbers);
                $mobile_numbers = array();
            }
            $row_count++;
        }
    }
    fclose($fp);
    $data_service->add_to_mobile_numbers_table($mobile_numbers_table, $mobile_numbers);
    $zip_file = "/$directory_name.zip";
    $logger->debug('Creating zipped file');
    Util::create_zip(Util::get_list_of_files($directory), $directory . $zip_file);

Answer 1

Following steps solved this issue:

• Change table engine from InnoDB to MyISAM

• disable the keys

• insert data

• re-enable the keys