Question

I have a method which returns the content of a webpage:

        private string FetchHTML(string sUrl, Encoding encoding)
        {
            System.Net.WebClient oClient = new System.Net.WebClient();
            oClient.Encoding = encoding;
            return System.Web.HttpUtility.HtmlDecode(oClient.DownloadString(sUrl));
        }

But when I try to load a link from livejournal (for instance, http://mos-jkh.livejournal.com/769579.html) then I am getting this exception at DownloadString:

The request was aborted: The operation has timed out.

Is it a known issue? Why doesn’t DownloadString work for some webpages and is there a solution for this? Or is there an alternative to DownloadString?

Answer 1

Some websites are smart enough to check whether the request is made by a browser or not. And when they detect that the request was done not with a browser they don’t respond. But it’s easy to fool them by simply sending the user agent info with the request. So the solution was adding one single line of code to the FetchHTML method:

    private string FetchHTML(string sUrl, Encoding encoding)
    {
        System.Net.WebClient oClient = new System.Net.WebClient();
        oClient.Encoding = encoding;
        // set the user agent to IE6
        oClient.Headers.Add("user-agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.0.3705;)");
        return System.Web.HttpUtility.HtmlDecode(oClient.DownloadString(sUrl));
    }

PS: To detect the issue I was using Fiddler instead of Wireshark which I’ve found too complex.