Question

I have a strange problem:

I am getting the html source from url using this:

string html;
HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(Url);
using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
{
   using (StreamReader reader = new StreamReader(response.GetResponseStream(), Encoding.UTF8))
   {
      html = reader.ReadToEnd();
      reader.Close();
   }
   response.Close();
}  

The page that I am requesting has css inline like this:

<span class="VL" style="display:inline-block;height:20px;width:0px;"></span>

But the html var value has only:

<span class="VL" style="display:inline-block;"></span>

Anyone knows why? I have tested with many enconders and using WebRequest and WebClient too, but doesn’t work too.

Answer 1

You might need to send a User Agent so that the site doesn’t think that you are a bot. Some sites don’t bother with CSS when requested from bots. Also the reading of the remote HTML could be simplified using a WebClient:

using (var client = new WebClient())
{
    client.Headers[HttpRequestHeader.UserAgent] = "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.4) Gecko/20100611 Firefox/3.6.4";
    string html = client.DownloadString(url);
}