Question

I’m trying to get google search results using HTML Agility Pack, but I am not getting the correct page. The page I am getting is Google’s main page,

This is the link the C#’s WebBrowser gives me :
https://www.google.com/#sclient=psy-ab&hl=en&site=&source=hp&q=hello+world&pbx=1&oq=hello+world&aq=f&aqi=&aql=&gs_sm=e&gs_upl=14782l16242l0l16965l11l7l0l0l0l0l0l0ll0l0&bav=on.2,or.r_gc.r_pw.,cf.osb&fp=5586f0ad4776d5b3&biw=268&bih=250

(if you try to paste it to a normal browser it will change to the URL below)

This is if I use a regular browser:
https://www.google.com/search?sclient=psy-ab&hl=en&site=&source=hp&q=hello+world&pbx=1&oq=hello+world&aq=f&aqi=&aql=&gs_sm=e&gs_upl=14782l16242l0l16965l11l7l0l0l0l0l0l0ll0l0&bav=on.2,or.r_gc.r_pw.,cf.osb&biw=268&bih=250&cad=h

private void GotoGoogleButton(object sender, EventArgs e)
{
    webBrowser1.Navigate(new Uri("https://www.google.com"));
}

private void getResultsButton(object sender, EventArgs e)
{
    url = webBrowser1.Url.ToString();
    start_parsing(url);
}

public void start_parsing(string URL_) 
{
    //This function is used to Parse Html source using Html Agility Pack into nodes, and take the important ones (mainly : a,form,iframe)

    HtmlAgilityPack.HtmlDocument document = webGet.Load(URL_);
    var metaTags = document.DocumentNode.SelectNodes("//a");

    // ...

    // Continue with parsing...
}

How do I get the search results instead of the home page?

Answer 1

I think you would be better served to look into Google’s RESTful API for search

It appears you can get an Atom feed containing search results.