Question

using the function below I am pulling rows from tables, encoding them, then putting them in csv format. I am wondering if there is an easier way to prevent high memory usage. I don’t want to have to rely on ini_set. I believe the memory consumption is caused from reading the temp file and gzipping it up. I’d love to be able to have a limit of 64mb ram to work with. Any ideas? Thanks!

function exportcsv($tables) {
    foreach ($tables as $k => $v) {
        $fh = fopen("php://temp", 'w');
        $sql = mysql_query("SELECT * FROM $v");
        while ($row = mysql_fetch_row($sql)) {
            $line = array();
            foreach ($row as $key => $vv) {
                $line[] = base64_encode($vv);
            }
            fputcsv($fh, $line, chr(9));
        }
        rewind($fh);
        $data = stream_get_contents($fh);
        $gzdata = gzencode($data, 6);
        $fp = fopen('sql/'.$v.'.csv.gz', 'w');
        fwrite($fp, $gzdata);
        fclose($fp);
        fclose($fh);
    }
}

Answer 1

untested, but hopefully you understand

function exportcsv($tables) {
    foreach ($tables as $k => $v) {
        $fh = fopen('compress.zlib://sql/' .$v. '.csv.gz', 'w');
        $sql = mysql_unbuffered_query("SELECT * FROM $v");
        while ($row = mysql_fetch_row($sql)) {
            fputcsv($fh, array_map('base64_encode', $row), chr(9));
        }
        fclose($fh);
        mysql_free_result($sql);
    }
}

edit-
points of interest are the use of mysql_unbuffered_query and use of php’s compression stream. regular mysql_query() buffers entire result set into memory. and using the compression stream gets rid of having to buffer the data yet again into php memory as a string before writing to a file.