Question

Possible Duplicate:
memory usage export from database to csv in php

I am exporting a large database and wanted to know the best way to make it have a low memory footprint.

I realize that I must do this in cycles that have a time limit and use low memory, fetching say 100 rows at a time and saving the information to the file then redirect to start a new cycle starting from where it finished on the previous cycle.

I am wondering whats the best way buffer the data to file and not run out of memory, at present the script gets all data as a string then saves to file when it has finished getting all rows from the database. Some times it runs out of memory, hence the need to fix.

Do I use fwrite() on the data fetched from the database instead of putting into a var or use a temp file? If I use a temp file when do I merge/rename into the backup file?

Basically what is the best way for the script to export the database data into a file without getting the error “Fatal Error: PHP Allowed Memory Size Exhausted”?

    function backup_tables($host, $user, $pass, $db, $tables = '*')
    {
            set_time_limit(0);

            $mysqli = new mysqli($host,$user,$pass, $db);
            if ($mysqli->connect_errno)
            {
                echo "Failed to connect to MySQL: (" . $mysqli->connect_errno . ") " . $mysqli->connect_error;
            }

            $return = '';

            $return .= "--\n";
            $return .= "-- Database: `$db`\n";
            $return .= "--\n\n";
            $return .= "-- --------------------------------------------------------\n\n";

            $numtypes = array(
                'tinyint', 
                'smallint',
                'mediumint',
                'int',
                'bigint',
                'float',
                'double',
                'decimal',
                'real'
            );

            // get all of the tables
            if ($tables == '*')
            {
                    $tables = array();
                    $result = $mysqli->query('SHOW TABLES');
                    while ($row = $result->fetch_row())
                    {
                            $tables[] = $row[0];
                    }

                    $result->close();
            }
            else
            {
                    $tables = is_array($tables) ? $tables : explode(',',$tables);
            }

            for ($z = 0; $z == 0; $z++)
            {
                echo $z.'<br>';

            // cycle through tables
            foreach ($tables as $table)
            {
                    //
                    $typesarr = array();
                    $result = $mysqli->query("SHOW COLUMNS FROM `".$table."`");

                    while ($row = $result->fetch_assoc())
                    {
                            $typesarr[] = $row;
                    }
                    $result->close();

                    #echo '<h2>'.$table.'</h2>';
                    #print("<pre>" . print_r($typesarr, true). "</pre>");

                    // table structure dump
                    $return .= "--\n";
                    $return .= "-- Table structure for table `$table`\n";
                    $return .= "--\n\n";                        
                    $return.= 'DROP TABLE IF EXISTS `'.$table.'`;'."\n\n";
                    $result = $mysqli->query("SHOW CREATE TABLE `".$table."`");
                    $row = $result->fetch_array();
                    $return.= $row[1].";\n\n";
                    $result->close();

                    // table data dump
                    $return .= "--\n";
                    $return .= "-- Dumping data for table `$table`\n";
                    $return .= "--\n\n";

                    $result = $mysqli->query("SELECT * FROM `".$table."`");
                    $num_fields = $result->field_count;

                    if ($result->num_rows > 0)
                    {
                            // put field names in array and into sql insert for dump
                            $fields_str = '';
                            $fields =  array();
                            $finfo = $result->fetch_fields();

                            foreach ($finfo as $val)
                            {
                                    $fields_str .= '`'.$val->name.'`, ';
                                    $fields[] = $val->name;
                            }                                

                            $fields_str = '('.rtrim($fields_str, ', ').')';
                            $return.= 'INSERT INTO `'.$table.'` '.$fields_str.' VALUES'."\n";

                            // cycle through fields and check if int for later use
                            for ($i = 0; $i < $num_fields; $i++) 
                            {
                                    // strip brackets from type
                                    $acttype = trim(preg_replace('/\s*\([^)]*\)/', '', $typesarr[$i]['Type']));
                                    $acttype = explode(' ', $acttype);

                                    // build array, is field int or not
                                    if (is_numeric(array_search($acttype[0], $numtypes)))
                                    {
                                            $numflag[$i] = 1;
                                    }
                                    else
                                    {
                                            $numflag[$i] = 0;        
                                    }
                            }
                    }  

                    $x = 0;
                    $num_rows = $result->num_rows;

                    // cycle through table rows
                    while($row = $result->fetch_row())
                    {
                            $x++;

                            // cycle through rows fields
                            for($j=0; $j<$num_fields; $j++) 
                            {          
                                    if (isset($row[$j]) and $j === 0) { $return .= '('; }

                                    // field data has value or not NULL
                                    if (isset($row[$j]))
                                    { 
                                            // field data dump (INT)
                                            if ($numflag[$j]==1)
                                            {
                                                    #echo '(INT) '. $fields[$j].' = '.$row[$j].'<br>';
                                                    $return.= $mysqli->real_escape_string($row[$j]);
                                            } 
                                            else
                                            {
                                                    // field data dump values (empty string, NULL and INT)
                                                    $return.= "'".$mysqli->real_escape_string($row[$j])."'";
                                                    #echo $fields[$j]." = '".$mysqli->real_escape_string($row[$j])."'<br>";
                                            }
                                    }
                                    else
                                    {
                                            // field data dump (NULL)
                                            if (is_null($row[$j]))
                                            {
                                                    $row[$j] = 'NULL';
                                                    #echo '(NULL) '. $fields[$j].' = '.$row[$j].'<br>';
                                                    $return.= $row[$j]; 
                                            }
                                            else
                                            {
                                                    // field data dump (empty string)
                                                    $return.= "''";
                                            }
                                    }

                                    if ($j<($num_fields-1)) { $return.= ', '; }
                            }

                            if ($x<$num_rows) { $return.= "),\n"; } else { $return .= ");\n"; }

                            #echo '<br>';
                    }
                    #echo 'Rows: '.$rows.'<br>';
                    #echo 'Iterations: '.$x.'<br>';
                    $return.="\n-- --------------------------------------------------------\n\n";
            }

            }

            $result->close();

            //save file
            $handle = fopen('/db-backup-'.time().'.sql','a');
            fwrite($handle,$return);
            fclose($handle);
    }

Examples welcome

Answer 1

Add MYSQLI_USE_RESULT as the resultmode parameter to the mysqli::query call in order to iterate over the results one row at a time, without transferring them all to PHP in one bunch. See also the documentation for mysqli::use-result.

Write each line directly to the output file, avoiding the $result variable. Combined with the above, this can lead to each row being fetched from the server and written to file, so PHP won’t have to store more than one row at a time.