Question

I have a simple function to scrape a web page that is suddenly getting a 503 error from the following code for a specific page:

try
{
  WebClient client = new WebClient();
  client.Proxy = null;
  string webhtml = client.DownloadString(strURL);
  client.Dispose();
  return (webhtml);
}

The url works when I enter it in IE or FF but it’s returning an error when this code runs. Other urls work fine. I’m guessing there is something different about the http request from .net than from either of the browsers. Is there any way to better emulate a browser-based http request?

Thanks.

Answer 1

Found the http headers I was looking for from this site. Very useful.

http://www.ericgiguere.com/tools/http-header-viewer.html