Question

Am trying to upload Do not call list xml file to MySql DB. The file size is 256MB. It gets uploaded but when its starts processing am looping the phone numbers and concatenating 1000 phone numbers in a variable. When it reaches the limit of 1000, am creating a INSERT query and executing the query. BUt the problem is after inserting 3,50,000 records it say “Fatal error: Allowed memory size of 2097152000 bytes exhausted (tried to allocate 24 bytes) in /var/domains/htdocs/aim/leads/dnc on line 1308)”.

Below is the code snippet.

                    if(($fp = fopen($destination,"r"))) {

                        $buffer = fgets($fp);
                        $objXml = new XML($buffer);
                        $lists = Set::reverse($objXml);

                        $buffer = fgets($fp);
                        $objXml = new XML($buffer);
                        $acc = Set::reverse($objXml);

                        try {
                            while(!feof($fp)){
                                $counter = 0;
                                $phonesNo = "";

                                while(!feof($fp) && ($buffer = fgets($fp)) && $counter != 1000){

                                    $objXml = "";
                                    $arrXml = "";

                                    $objXml = new XML($buffer);

                                    $arrXml = Set::reverse($objXml);


                                    $counter++;
                                    if(isset($arrXml['Ph']['val']))
                                        $phonesNo .= "(".$acc['Ac']['val'].$arrXml['Ph']['val']."),";

                                }

                                $phonesNo = substr($phonesNo,0,-1);
                                $sql = "INSERT INTO do_not_call_lists (`phone`) VALUES " .$phonesNo;

                                unset($phonesNo);
                                $this->Lead->query($sql);
                            }
                        }catch (Exception $ex){
                            trigger_error("ERROR: ". $ex->getMessage(). "  Trace: ".$ex->getTrace());
                        }

                        $this->Session->setFlash('File imported successfully.','default',array('class'=>'sucessMsg'));

                     }

Thanks a lot in advance.

Answer 1

I think what he means is not how to extend the limits, but why the hell is this allocating 2GB of RAM?

From the looks of it, I’m assuming you’re using simplexml to load the data. SimpleXML is a memory overkill and it can’t be used for large files without exactly this happening. You should instead try using:

1. event based SAX parser (less memory usage than the 2nd one, but much harder to operate)
2. pull based XMLReader

FYI, I’ve been using SAX to load about 200MB XML file with peak memory usage about 5MB, so yes, the memory consumption is uncomparable.