I’m building a portal that lists certain products and automatically gets the prices from product pages of listed vendors. To get the URL for the product page on a vendor’s website, I’ve been using Google search API and it’s been working great – the first result is invariably the page of the product. However, now I’m getting errors saying that Google has blocked my website (actually my develoment machine’s IP) from the API because I’ve been making automated requests such as scraping (the only item that applies).
Fine, Google can go jump off a cliff, but… how do product portals generally get URLs for thewir products? I can enter the URLs manually but that can be a problem if the vendor’s website changes the URL scheme somehow. I obviously need an automated way to do this.
I’m making no more than 50-60 requests per day so I don’t get what Google wants. Do they want money?
First, they want you to use one of their APIs, not scrape their web page directly. Their custom search API is documented here. Once you register they’ll give you an API key. You can get results in JSON format by requesting
Second, they do like money, but you might be okay. You’re allowed 100 searches per day for free; beyond that you’re you’re going to be charged $5 per thousand searches.