Question

I need to analyze one particular page on google and I have a direct lik to that page. When I paste that link to IE, it works fine.

Now I tried to automate this process

WebClient w = new WebClient();
                string s = w.DownloadString("https://www.google.com/#q=" + MYSEARCHSTRING + "&start=0");

But it is not loading the page properly into ‘s’

Answer 1

When loading a url with code you cannot use the ‘#’ character in it, because that is a Fragment Identifier and google now runs the search using javascript so the browser doesn’t request each page anymore.

To do this you need to use the older search syntax: https://www.google.com/search?q=

A better solution might be to use the API to search: Custom Search, using that would be nicer on your parsing of the results and nicer on Google.

One thing to be mindful of is to make sure how you are using the results fits into the Terms of Service, using the API should help avoid being able to abuse it as well, I have not looked at the search terms ever so I’m uncertain what restrictions might be there.