Question

I have used this code for extracting urls from web page.But in the line of ‘foreach’ it is showing

Object reference not set to an instance of an object

exception. What is the problem? how can i correct that?

WebClient client = new WebClient();
string url = "http://www.google.co.in/search?hl=en&q=java&start=10&sa=N";
string source = client.DownloadString(url);
HtmlDocument doc = new HtmlDocument();
doc.LoadHtml(source);
foreach (HtmlNode link in doc.DocumentNode.SelectNodes("//a[@href and @rel='nofollow']"))
{
    Console.WriteLine(link.Attributes["href"].Value);
}

Answer 1

First, you should look up the NullReferenceException in the documentation. It says

The exception that is thrown when there is an attempt to dereference a null object reference.

This means you did the equivalent of

SomeClass reference = null;
reference.Method(); // or reference.Property;

Next, look at the line of code that has the error and figure out what you are derefencing:

foreach (HtmlNode link in doc.DocumentNode.SelectNodes("//a[@href and @rel='nofollow']"))

• doc.DocumentNode
• doc.DocumentNode.SelectNodes

So either doc is null, or doc.DocumentNodes is null. Since you just assigned a new instance of HtmlDocument to doc, doc can’t be the problem. That implies that you loaded an empty document, such that there is no doc.DocumentNode.

Check before the loop to see if doc.DocumentNode is null.