Question

The problem:
I want to scrap some data from certain webpage (I have administrative access) and to store some information in db for later analysis.
Sounds easy, right?
I’ve decided to make simple console prototype and code look something like this:

string uri =  @"http://s7.iqstreaming.com:8044/admin.cgi";
HttpWebRequest request = WebRequest.Create(uri) as HttpWebRequest;

if(request == null)
{
     Console.WriteLine(":( This shouldn't happen!");
     Console.ReadKey();
}

request.ContentType = @"text/html";
request.Accept = @"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
request.Credentials = new NetworkCredential("myID", "myPass");

using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
     StreamReader reader = new StreamReader( response.GetResponseStream());

     while (!reader.EndOfStream)
     {
         Console.WriteLine(reader.ReadLine());
     }

     reader.Close();
     response.Close();
}

This code works on most other sites, but here I get errors 404 (most of the time), 502 or timeout.
I’ve consulted with Firebug (I’ve took Accept and compression info from there) but to no avail.
Using Win-forms and webBrowser control as an alternative is not an option (at least for now).

P.S.
Same thing happens when I try to get HTML from http://s7.iqstreaming.com:8044/index.html (doesn’t need credentials).

Answer 1

I think the problem is related with User-Agent.
This may solve it

request.UserAgent="Mozilla/5.0 (Windows NT 5.1) AppleWebKit/535.11 (KHTML, like Gecko) Chrome/17.0.963.78 Safari/535.11";