Question

I have a script that compiles a list of items that can be very long – just compiling this list can take 10-15 minutes, but that is acceptable. When I incorporate a function that iterates through the list and inserts them all into a mySQL table, that time is increased about about 50%. I was wondering if there was a faster way of serializing this data? Should i explore CSV or something else? Or can I optimize my code to do this faster:

    private function toDB(){
    $sql[] = "DROP TABLE IF EXISTS checklisttest";$sql[] = "CREATE TABLE checklisttest (
  Incident varchar(12) NOT NULL,
  TestID mediumint(9) NOT NULL AUTO_INCREMENT,
  Element varchar(12) NOT NULL,
  Name varchar(128) NOT NULL,
  Code varchar(512) NOT NULL,
  Expected varchar(512) NOT NULL,
  Actual varchar(512) NOT NULL,
  AutoVerifyResult varchar(32) NOT NULL,
  QAResult varchar(32) DEFAULT NULL,
  Comments text,
  PRIMARY KEY (TestID)
)";

    //iterate through the records $this->records[10001] -- There can be anywhere from 100 - 300 records
    foreach($this->records as $inc => $record){
        //iterate through the element ids $this->records[10001][E02_04]
        foreach($this->records[$inc]["Elements"] as $elementID => $element){
            //iterate through the element ids $this->records[10001][E02_04][1] --There can be anywhere from 150 - 350 elements per record.
            foreach($element as $key => $val){
                $sql[] = "
INSERT INTO `checklistTest` VALUES (\"$inc\",NULL,\"$elementID\",\"$val[name]\",\"$val[code]\",\"$val[expected]\",\"$val[actual]\",\"$val[match]\",\"$val[QAResult]\",NULL)";
            }
        }
    }
    foreach($sql as $key => $val){
        mysql_select_db("new",$GLOBALS['local']);
        mysql_query($val,$GLOBALS['local']) or die(mysql_error());
    }
    //echo "<textarea style='width:100%;height:400px'>$sql</textarea>";
    //mysql_select_db("new",$GLOBALS['local']);
    //mysql_query($sql,$GLOBALS['local']) or die(mysql_error());
}

There must be a better way to go about doing this, I just don’t have much experience performing a lot of queries like this – usually they are just one-and-done for me. Thanks for the help.

thanks for the answers, I posted my solution in a comment to the accepted answer.

Answer 1

Insert of firing multiple insert statements you can combine them in to one query as below –

//iterate through the records $this->records[10001] -- There can be anywhere from 100 - 300 records

$sql = “INSERT INTO checklistTest VALUES “;

    foreach($this->records as $inc => $record){
        //iterate through the element ids $this->records[10001][E02_04]

        foreach($this->records[$inc]["Elements"] as $elementID => $element){

            //iterate through the element ids $this->records[10001][E02_04][1]--There can be anywhere from 150 - 350 elements per record.
            foreach($element as $key => $val){
                $sql.= "(\"$inc\",NULL,\"$elementID\",\"$val[name]\",\"$val[code]\",\"$val[expected]\",\"$val[actual]\",\"$val[match]\",\"$val[QAResult]\",NULL),";
            }
        }
    }

Note: Now here remove the last comma. ie for the last value in array too a comma will be appended at the end do remove that else you will get database error.

Now because of above merging of insert queries you have to query your database only once which improves your query perfomance a lot.

mysql_select_db(“new”,$GLOBALS[‘local’]);

mysql_query($sql,$GLOBALS[‘local’]) or die(mysql_error());

//echo "<textarea style='width:100%;height:400px'>$sql</textarea>";
//mysql_select_db("new",$GLOBALS['local']);
//mysql_query($sql,$GLOBALS['local']) or die(mysql_error());

}

Your can refer below links for detailed documentation of this method –

http://www.brainbell.com/tutorials/MySQL/Inserting_Multiple_Rows.htm

http://dev.mysql.com/doc/refman/5.5/en/insert.html