Question

I am working on a Microsoft .NET Application in C# for Web Harvesting, Web Scraping, Web Data Extraction, Screen Scraping, etc. Whatever you want to call it. For parsing HTML, I’m attempting to incorporate HTML Agility Pack but it’s not as easy as I thought it would be. I have included some specifications and images of what I have so far and was hoping to get your opinions on how I could proceed. basically, I want to do something similar to the layout used in Visual Web Ripper but I have no idea how they do it… Any ideas?

Specifications:

My goal is to make a very user friendly point-and-click application for downloading data and images from the web. I would like to load HTML pages using the web browser, and output the parsed data and image links into the text box. The user can specify which HTML tags they want and then download the data into the grid. Finally, export the data into whatever format they need.

I’m trying to use HTML Agility Pack to load the HTML on the webpage and display it in the textbox.

// Load Web Browser
private void Form6_Load(object sender, EventArgs e)
{
    // Navigate to webpage
    webBrowser.Navigate("http://www.webopedia.com/TERM/H/HTML.html");
    // Save URL to memory
    SiteMemoryArray[count] = urlTextBox.Text; 
    // Load HTML from webBrowser
    HtmlWindow window = webBrowser.Document.Window; 
    string str = window.Document.Body.OuterHtml;
    // Extract tags using HtmlAgilityPack and display in textbox
    HtmlAgilityPack.HtmlDocument HtmlDoc = new HtmlAgilityPack.HtmlDocument();
    HtmlDoc.LoadHtml(str);
    HtmlAgilityPack.HtmlNodeCollection Nodes =
        HtmlDoc.DocumentNode.SelectNodes("//a");
    foreach (HtmlAgilityPack.HtmlNode Node in Nodes)
    {
        textBox2.Text += Node.OuterHtml + "\r\n";
    }
}

Using:

HtmlWindow window = webBrowser.Document.Window;

I get the error: Object reference not set to an instance of an object.

Answer 1

You might not have the page load completed when you are referencing the browser window. You can have the browser control fire the navigationcomplete event when it is done. See this SO answer for an example: C# how to wait for a webpage to finish loading before continuing