Question

I needed a method that gets a link to a page and returns the title of this page.

I used WebClient –

        var webClient = new WebClient();
        var htmlString = webClient.DownloadString(_link);

It works well, but it fails with encoding of foreign languages. I’m getting question marks and weird characters instead of the text I need.

Is there a generic way to identify the encoding of the page and use it? I need it to support most of the encodings if not all.

Answer 1

Using the HtmlAgilityPack you can do something like this

using (WebClient client = new WebClient())
using (var read = client.OpenRead("http://your.com"))
{
    HtmlDocument doc = new HtmlDocument();
    doc.Load(read, true); // true = get encoding from byte order masks
    // process doc, extract title
    var title = doc.DocumentNode.SelectSingleNode("//title").InnerText;
}