Question

Possible Duplicate:
cURL Mult Simultaneous Requests (domain check)

I’m trying to check to see if a website exists. (if it responds that’s good enough) The issue is my array of domains is 20,000 and I’m trying to speed up the process as much as possible.

I’ve done some research and come across this page which details simultaneous cURL requests ->
http://www.phpied.com/simultaneuos-http-requests-in-php-with-curl/

I also found this page which seems be a good way of checking if a domain webpage is up -> http://www.wrichards.com/blog/2009/05/php-check-if-a-url-exists-with-curl/

Any ideas on how to quickly check 20,000 domains to see if they are up?

Answer 1

check out RollingCurl

It allows you to execute multiple curl requests.
Here is an example:

    require 'curl/RollingCurl.php';
    require 'curl/RollingCurlGroup.php';


    $rc = new RollingCurl('handle_response');
            $rc->window_size = 2;




            foreach($domain_array as $domain => $value)
            {






                 $request = new RollingCurlRequest($value);

                // echo $temp . "\n";


                    $rc->add($request);




            }

            $rc->execute();




    function handle_response($response, $info)
    {

            if($info['http_code'] === 200)
            {
                // site exists handle response data
            }

    }