Question

I need to read webpage and store its content in string for further processing.
Sounds simply but I have problem with cookies support.

Opened page says I need browser supporting cookies (or turned on).
I’ve made method trying do that via httpWebRequest – which normally works to me but I’ve come to a standstill with those unfortunate cookies…

Any idea how to make it working?

Here is my method:

string ReadHtml (string address, string encoding) {
   Uri url = new Uri(address);

   CookieContainer cookieContainer  = new CookieContainer();
            HttpWebRequest httpWebRequest  =     (HttpWebRequest)HttpWebRequest.Create(url);
   httpWebRequest.AllowAutoRedirect = true;
            httpWebRequest.KeepAlive   = true;
            httpWebRequest.CookieContainer  = cookieContainer;
            httpWebRequest.UserAgent   = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)";
            httpWebRequest.Method    = "GET";
            HttpWebResponse webResponse = (HttpWebResponse)httpWebRequest.GetResponse();

   // Code Page
   Encoding enc = Encoding.GetEncoding(encoding);  

   // Read content
   StreamReader loResponseStream = new     StreamReader(webResponse.GetResponseStream(),enc);
   string   lcHtml    = loResponseStream.ReadToEnd();

   webResponse.Close();
   loResponseStream.Close();

   return lcHtml;

  }

Answer 1

The page (until you r making request) sends you a cookie and at another request this page is trying to read thatcookie. When you don’t provide cookie, then page think that your browser desn’t support that mechanism.

I recommend you to:

1. Open Fiddler application
2. Check what cookies are saved (trying to save)

The solution is to provide back to page exacly that cookie, that this page sends you.