Question

I need a .php script that will download a lot of images from another site. The images are thumbs – each has about 20KB size.
I have worked on my own script, but sadly it just lags my server and nearly kills it forcing me to restart it.

There are about 100 pictures or more per execution, .jpg files, ~20KB / file.

My script:

$count = 0;
foreach ($files as $file) {
$count++;
$url = $file;
$dl_place = '/home/lulz/'.$count.'.jpg';

$ch = curl_init($dl);
$fp = fopen($path, 'wb');
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, 0);
curl_exec($ch);
curl_close($ch);
fclose($fp);
}

As you see I am using curl, but I am willing to use anything if it just works better than it is now.

Answer 1

Chances are, what is slowing down is the time it takes to set up all of these requests. You should consider Parallel cURL to download multiple at a time. Source code: https://github.com/petewarden/ParallelCurl/blob/master/parallelcurl.php

$pc->startRequest('http://www.whatever.com/someimage.jpg', 'your_callback_function');

I have also found that with library, you can use anonymous functions instead of the name of a function in your callback. I use this to call another function with an ID number, for example.

$requestid=37;
$pc->startRequest(
    $url, 
    function($content, $url, $ch, $search) use $requestid {
        yourRealCallback($content, $url, $ch, $search, $requestid);
    }
);

This utilizes an anonymous function with closure so that if you are searching a DB of URLs, you can get the resulting ID (that you specify in a for loop or something… hard-coded to ’37’ here for demonstration).