Question

Currently, you can import a spreadsheet (contains a list of websites) into a database. You are able to click a button and it retrieves the number of pages that Google has indexed for each domain in the database.

It works fine up until about ~400 requests and then my requests come back with nothing. Not sure what I am doing incorrectly or how to make this work – any ideas?

try {
        require('db.php');
        $conn = new PDO ( "mysql:host=localhost;dbname=" . $database, $user, $pass );
        $conn->setAttribute( PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION );
        $stmt = $conn->query('SELECT `url` FROM domains');
        $stmt->setFetchMode(PDO::FETCH_ASSOC);
        while($row = $stmt->fetch()){
            $id = $row['id'];
            $url = $row['url'];
            $pages = getIndexedPageCount($url);
            if ( $pages > 0 ) {
                $status = 1;
            } else {
                $status = 0;
            }
            $prep = $conn->prepare("
            UPDATE `domains`
                SET
                    url = :url,
                    pages = :pages,
                    status = :status
                WHERE url = :url
            ");
            $prep->execute(array(
                ':url'      =>  $url,
                ':pages'    =>  $pages,
                ':status'   =>  $status
            ));
        }
    } catch ( PDOException $e ) {
        echo 'Error: ' . $e->getMessage();
        exit;
    }


    function getIndexedPageCount($domain) {

    // remove http:// and https://
    if ( strpos( $domain, 'http:' ) == 0 ) { $domain = str_replace('http://', '', $domain); }
    if ( strpos( $domain, 'https:' ) == 0 ) { $domain = str_replace('https://', '', $domain); }

        $content = file_get_contents('https://www.googleapis.com/customsearch/v1?key={{Removed API Key}}&q=site:' . $domain);

        if (strlen($content) > 1) {
            $data = json_decode($content);
            return intval($data->searchInformation->totalResults);
        }

        else {
            echo "Error: URL does not exist.";
        }
    }

Any help would be greatly appreciated; if anyone has an idea I’ll try and make it work!

Answer 1

There is a limit to the number of requests you can send to Google.

See here for details https://developers.google.com/console/help/#monitoringandfiltering

Here is the API console: https://code.google.com/apis/console

In the console currently it says for Custom Search API there is a Courtesy limit: 100 requests/day

Any usage beyond the free usage quota will fail if you are not signed up for billing. Once you have enabled billing, you will continue to receive 100 free queries per day. However, you will be billed for all additional requests at the rate of $5 per 1000 queries, for up to 10,000 queries per day.