Question

Im working on a little hobby project. I already have written the code to get a url, download the header and return the mime type / content type.

However, the step before this is the one im stuck on – i need to retrieve the contents of all the urls on the page based inside a tag, and in quotes i.e.

...
<link rel='shortcut icon' href="/static/favicon.ico" type="image/x-icon" />
...

Would find the favicon link.

Is there anything helpful in the .net library or is this going to have to be a case for regex?

Answer 1

I’d look at using the Html Agility Pack.

Here’s an example straight from their examples page on how to find all the links in a page:

 HtmlWeb hw = new HtmlWeb();
 HtmlDocument doc = hw.Load(/* url */);
 foreach(HtmlNode link in doc.DocumentNode.SelectNodes("//a[@href]"))
 {

 }