Question

I want to programmatically retrieve Google Search Results for the purpose of finding where a specific domain ranks in the search results pages.

HOWEVER, I do NOT want to simply crawl search result pages because I’m expecting high volume and needing to do this frequently, and this is judged as abuse by Google if I understand correctly?

Most scripts/classes I’ve been able to find try to parse the HTML pages, there’s gotta be a better way.

Is there an API to get google results?
Any ideas?

Thanks!

Answer 1

first you should understand something:
there is not “a” ranking. the SERP you see when googling your keywords is not the same SERP other people see when googleing your keywords. they are a sh*tload of “personalization” factors (location, cookie enabled, instant search, day time, previous searches, web history, datacenter, ….) that come into account of where something ranks. for some popular keywords the top 3 are kinda static, 5 to 10 in a flux, after ten it gets really really fuzzy, after 20 its like throwing a dice.

and that is just the crawl the google serps approach.

it gets worse with the web search api (deprecated but working) or the custom search api (== crap == d*ckmove by google).

so whatever you do, you will always just get a near meaningless snapshot of the google results.

and no, there is no other offical API.

that was the bad news, now the good news … if you worry about your own domain, just go to “google webmaster tools” and click on “search queries”. that’s the best information you can get (it’s still fuzzy, but it’s what you get found for, where you rank on average). or you can apply some specialized google anlaytics filter to check the rank postion of google referred traffic.

if you want to analyse your competition, well there are a lot of search marketing companies which sell exactly that kind of service (most of them are specialized per market, i.e. in germany it’s sistrix, there are a sh*tload of such companies in the us).

but as i said before: the data is a meaningless snapshot and most of the time just not actionable.