Home/ Questions/Q 7803367
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-02T01:33:34+00:00

I have a bit of a problem when I try to take a huge

  • 0

I have a bit of a problem when I try to take a huge amount of data from a mysql table to a redis database. Anyway I’m getting the error “MySQL server has gone away” after a while and I have no idea why..

EDIT:

OR when I use the commented code that breaks the loop it just goes “finished” when it isn’t finished.

This is the php code I use (runned by php-cli):

<?php
require 'Predis/Autoloader.php';
Predis\Autoloader::register();

mysql_connect('localhost', 'root', 'notcorrect') or die(mysql_error());
mysql_select_db('database_that_i_use') or die(mysql_error());
$redis = new Predis\Client();

//starting on 0 but had to edit this when it crashed :(
for($i = 3410000; $i<999999999999; $i += 50000) {
        echo "Query from $i to  " . ($i + 50000) . ", please wait...\n";
        $query = mysql_unbuffered_query('SELECT * FROM table LIMIT ' . $i . ', 50000')or die(mysql_error());
        // This was code I used before, but for some reason it got valid when it wasn't supposed to. 
        /*if(mysql_num_rows($query) == 0) {
                echo "Script finished!\n";
                break;
        }*/
        while($r = mysql_fetch_assoc($query)) {
                $a = array('campaign_id' => $r['campaign_id'],
                           'criteria_id' => $r['criteria_id'],
                           'date_added' => $r['date_added'],
                );

                $redis->hmset($r['user_id'], $a);
                unset($a);
                usleep(10);
        }
        echo "Query completed for 50000 rows..\n";
        sleep(2);
}



unset($redis);
?>

My question is how to do this better, I have seriously no idea why it crashes. My server is pretty old and slow and maybe can’t handle this large amount of data? This is just a testserver before we switch to real production.

Worth to notice is that the script ran fine for maybe half an hour and it may be the limit statement that makes it very slow when the number get high? Is there then an easier way to do this? I need to transfer all the data today! 🙂

Thanks in advance.

EDIT: running example:

Query from 3410000 to  3460000, please wait...
Query completed for 50000 rows..
Query from 3460000 to  3510000, please wait...
Query completed for 50000 rows..
Query from 3510000 to  3560000, please wait...
Query completed for 50000 rows..
Query from 3560000 to  3610000, please wait...
MySQL server has gone away

EDIT:

The table consist of ~5 million rows of data and is approx. 800 MB in size.
But I need to do similar things for even larger tables later on..

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    First, you may want to use another script language. Perl, Python, Ruby, anything is better than PHP to run this kind of scripts.

    I cannot comment on why the mysql connection is lost, but to get better performance you need to try to eliminate as many roundtrips as you can with the mysql server and the redis server.

    It means:

    • you should not use unbuffered queries but buffered ones (provided LIMIT is used in the query)

    OR

    • you should not iterate on the mysql query using LIMIT since you get a quadratic complexity while it should be only linear. I don’t know if it can be avoided in PHP though.

    • you should pipeline the commands you sent to Redis

    Here is an example of pipelining with Predis:
    https://github.com/nrk/predis/blob/v0.7/examples/PipelineContext.php

    Actually, if I really had to use PHP for this, I would export the mysql data in a text file (using “select into outfile” for instance), and then read the file and use pipelining to push data to Redis.

    • 0