Question

I have a do/while loop that goes over database rows. Because it runs many days at the time processing 100000s of rows, memory consumption is important to keep in check or it will crash. Right now every iteration adds about 4kb to script’s memory usage. I’m using memory_get_usage() to monitor the usage.

I unset every variable used in the loop first thing in each iteration so I really don’t know what else I could do. My guess is that do/while gathers some data with each iteration and this is what consumes the 4kb of memory. I know 4kb doesn’t sound like much but it soon starts to add up when you have 100000s of iterations.

Can somebody suggest another way of going through large amount of database rows or how to somehow eliminate this “memory leak”?

edit
Here’s the UPDATED loop code. Above it is just few require_once()s.

$URLs = new URLs_url(db());
$c = new Curl;
$c->headers = 1;
$c->timeout = 60;
$c->getinfo = true;
$c->follow = 0;
$c->save_cookies = false;

do {
    // Get url that hasn't been checked for a week
    $urls = null;

    // Check week old
    $urls = $URLs->all($where)->limit(10);

    foreach($urls as $url) {
        #echo date("d/m/Y h:i").' | Checking '.$url->url.' | db http_code: '.$url->http_code;

        // Get http code    
        $c->url = $url->url;
        $data = $c->get();

        #echo ' - new http_code: '.$data['http_code'];

        // Save info
        $url->http_code = $data['http_code'];
        $url->lastchecked = time();
        $URLs->save($url);
        $url = null;
        #unset($c);
        $data = null;
        #echo "\n".memory_get_usage().' | ';
        echo "\nInner loop memory usage: ".memory_get_usage();
    }
    echo "\nOuter loop memory usage: ".memory_get_usage();

} while($urls);

Some logs how memory consumption behaves in both loops:

Inner loop memory usage: 611080
Inner loop memory usage: 612452
Inner loop memory usage: 613788
Inner loop memory usage: 615124
Inner loop memory usage: 616460
Inner loop memory usage: 617796
Inner loop memory usage: 619132
Inner loop memory usage: 620500
Inner loop memory usage: 621836
Inner loop memory usage: 623172
Outer loop memory usage: 545240
Inner loop memory usage: 630680
Inner loop memory usage: 632016
Inner loop memory usage: 633352
Inner loop memory usage: 634688
Inner loop memory usage: 636088
Inner loop memory usage: 637424
Inner loop memory usage: 638760
Inner loop memory usage: 640096
Inner loop memory usage: 641432
Inner loop memory usage: 642768
Outer loop memory usage: 556392
Inner loop memory usage: 640416
Inner loop memory usage: 641752
Inner loop memory usage: 643088
Inner loop memory usage: 644424
Inner loop memory usage: 645760
Inner loop memory usage: 647096
Inner loop memory usage: 648432
Inner loop memory usage: 649768
Inner loop memory usage: 651104
Inner loop memory usage: 652568
Outer loop memory usage: 567608
Inner loop memory usage: 645924
Inner loop memory usage: 647260
Inner loop memory usage: 648596
Inner loop memory usage: 649932
Inner loop memory usage: 651268
Inner loop memory usage: 652604
Inner loop memory usage: 653940
Inner loop memory usage: 655276
Inner loop memory usage: 656624
Inner loop memory usage: 657960
Outer loop memory usage: 578732

Answer 1

This bit should probably happen only once, before the loop:

$c = new Curl;
$c->headers = 1;
$c->timeout = 60;
...
$c->getinfo = true;
$c->follow = 0;
$c->save_cookies = false;

Edit: Oh, the entire thing is wrapped in a do/while loop. /facepalm

Edit 2: There’s also this important bit:

unset($class_object) does not release
resources allocated by the object. If
used in loops, which create and
destroy objects, that might easily
lead to a resource problem. Explicitly
call the destructor to circumvent the
problem.

http://www.php.net/manual/en/function.unset.php#98692

Edit 3:

What is this? Can’t this be moved outside of the loop somehow?

$URLs = new URLs_url(db());

Edit 4:

Try removing these lines, for now.

    $url->http_code = $data['http_code'];
    $url->lastchecked = time();
    $URLs->save($url);