Question

how can i download a web page from my web app, then reading “title” and “description” metatag ?
Like a web crawler, but in Asp.net and called by ..an asp.net web page ?

Thanks!

Answer 1

You can do a screen scrape of an external URL in .NET using the WebClient class, which you’ll find in the System.Net namespace. Use the DownloadData method to download the content from a specified URL. The downloaded data comes down as a byte array, but you can convert this to a string.

The following snippet shows how to use WebClient to grab the HTML from my blog’s homepage, http://scottonwriting.net/sowblog/default.aspx:

// Create a new WebClient instance.
WebClient myWebClient = new WebClient();

// Download the markup from 
byte[] myDataBuffer = myWebClient.DownloadData("http://scottonwriting.net/sowblog/default.aspx");

// Convert the downloaded data into a string
string markup = Encoding.ASCII.GetString(myDataBuffer);

Once you have the markup you can use regular expressions or string searching methods to pick out the markup of interest.