Question

I have been trying to use the htmlagilitypack through a proxy and I see that I get some unpredictable behavior.

How do you add credentials to htmlagilitypack so that it will be able to scrape web pages through a proxy?

Answer 1

Here is what I usually do:

HttpWebRequest request = (HttpWebRequest) HttpWebRequest.Create(url);
...
HttpWebResponse response = (HttpWebResponse) request.GetResponse();

using (var reader = new StreamReader(response.GetResponseStream(), Encoding.UTF8)) {
    var doc = new HtmlDocument();
    doc.Load(reader.BaseStream);

    //Use (or return) the HtmlDocument 'doc' here.
}

You could encapsulate this code on a method that given a url, returns a HtmlDocument object.