I’m trying to program a simple search in google via C# that would run a query of my choice and retrieve the first 50 links. After thoroughly searching for a similar tool\correct API I realized that most of them are obsolete. My first try was to create a “simple HttpWebRequest” and scan the received WebResponse for “href=” which turned out to be not rewarding at all (redundancy) and very frustrating. I do have a Google API but I’m not sure how to use it for this purpose, although I know that there is an 1000 limit per day.
Gil
If you’re going this route you should use HtmlAgility pack for your parsing. However, a better approach would be to use Google’s API. See this post i need to know which of my url is indexed on google
As for some code for using HtmlAgility pack, I have a post on my blog
Finding links on a Web page