Question

So I keep running into the same error… I’ve searched for hours to try to find a resolution, but I just can’t seem to find the missing piece. Lots of other people asking about error 7 on stack overflow, but none that were similar to my scenario.

Basically, I’m using cURL to download images being sent through an XML feed. My entire script works, everything runs, the function I’ve written below even downloads thousands of images (upwards to the 3000 range sometimes).

I guess my question is, why, after downloading 3000 images would it just not connect?

function downloadImage($location, $imagesPath, $imageName) {

    //Location fix
    $location = str_replace(" ", "%20", $location);

    $url  = $location;
    $path = $imagesPath . $imageName;
    $fp = fopen($path, 'w');    

    $ch = curl_init($url);
    curl_setopt($ch, CURLOPT_FILE, $fp);
    curl_setopt($ch, CURLOPT_FRESH_CONNECT, true);
    curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 0); //Wait indefinitately      
    curl_setopt($ch, CURLOPT_DNS_USE_GLOBAL_CACHE, false);

    $data = curl_exec($ch); 
    if ($data === false) {
      echo "DownloadImage cURL failed 1: (" . curl_errno($ch) . ") " . curl_error($ch) . "<br/>";
      //exit;
    }           
    curl_close($ch);


    fclose($fp);        

}

Answer 1

So in order to get this to work, I had to put a bottleneck on the function to slow it down a bit. As Andrewsi suggested, the remote site was cutting me off for downloading images too quickly. In order to bottleneck the function, I FTP’d each image to a remote server (since this was required anyway).

Final function looks like this:

function downloadImage($location, $imagesPath, $imageName, $ch3, $feedFTPinfo) {

//Location fix
//$location = str_replace(" ", "%20", $location);   

$url  = $location;
$path = $imagesPath . $imageName;

$fp = fopen($path, 'w');    
$ch2 = curl_init(); // Initiate cURL for downloading images

//Setup the cURL options for the second handle ($ch2)
curl_setopt($ch2, CURLOPT_URL, $url);
curl_setopt($ch2, CURLOPT_FILE, $fp);   
curl_setopt($ch2, CURLOPT_CONNECTTIMEOUT, 0); //Wait indefinitately     
curl_setopt($ch2, CURLOPT_FOLLOWLOCATION, true);

//Execute the cURL session
$data2 = curl_exec($ch2); 

//Resize Images
$image = new SimpleImage();
$image->load($url); 
$imageNameSm = str_replace(".jpg", "", $imageName);
$imageNameSm = $imageNameSm."_sm2.jpg"; 
$image->resizeToWidth(120);
$image->save($imagesPath . $imageNameSm);   
$smPath = $imagesPath . $imageNameSm;

//Find out if there were any issues
if ($data2 === false) {
  echo "DownloadImage cURL failed 1: (" . curl_errno($ch2) . ") " . curl_error($ch2) . "<br/>";
  //exit;
} else {
    //There weren't any issues downloading the file, move it to the speficifed ftp server

    if (!empty($feedFTPinfo)) {

        $localfile = $path;
        $fp = fopen($localfile, 'r');

        //Setup the options for the 3rd handle
        curl_setopt($ch3, CURLOPT_URL, $feedFTPinfo.$imageName);
        curl_setopt($ch3, CURLOPT_CONNECTTIMEOUT, 0); //Wait indefinitately 
        curl_setopt($ch3, CURLOPT_UPLOAD, 1);
        curl_setopt($ch3, CURLOPT_INFILE, $fp);
        curl_setopt($ch3, CURLOPT_INFILESIZE, filesize($localfile));

        //Execute the 3rd cURL handle
        $data3 = curl_exec($ch3);

        //Find out if there were any issues with the execution
        if ($data3 === false) {
            echo "Uploading the image via FTP failed: (" . curl_errno($ch3) . ") " . curl_error($ch3) . "<br/>";
            //exit;
        }

        $localfile = $smPath;
        $fp = fopen($localfile, 'r');

        //Setup the options for the 3rd handle
        curl_setopt($ch3, CURLOPT_URL, $feedFTPinfo.$imageNameSm);
        curl_setopt($ch3, CURLOPT_CONNECTTIMEOUT, 0); //Wait indefinitately 
        curl_setopt($ch3, CURLOPT_UPLOAD, 1);
        curl_setopt($ch3, CURLOPT_INFILE, $fp);
        curl_setopt($ch3, CURLOPT_INFILESIZE, filesize($localfile));

        //Execute the 3rd cURL handle
        $data3 = curl_exec($ch3);

        //Find out if there were any issues with the execution
        if ($data3 === false) {
            echo "Uploading the small image via FTP failed: (" . curl_errno($ch3) . ") " . curl_error($ch3) . "<br/>";
            //exit;
        }           




    }


}

curl_close($ch2); //Close the cURL handle that downloads images
fclose($fp);
}

If you don’t need to ftp somewhere to create the bottleneck, you could use php’s sleep (seconds) or usleep (microseconds) functions within your downloadImage function to create a similar bottleneck.

• sleep documentation:
  http://php.net/manual/en/function.sleep.php
• usleep documentation:
  http://php.net/manual/en/function.usleep.php

Hope this theory helps someone else out.