Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
I am trying to crawl a web site and fetch its pages’ data. It all went well till one point where I started to get different data than the one coming in the view-source (the one I have expected). I am using html-agility-pack but also tried using HttpWebRequest, in order to change all it’s headers, but no luck. I know it isn’t a JavaScript trick because of the data in the view source. I hope someone has an idea to a possible explanation or solution.
There are a lot of factors that could possibly affect the html sent to you by the server. I suggest you use a tool such as fiddler to inspect the request headers more closely.
From my experience the usual culprits in such problems are cookies and session information stored as hidden fields in the document and sent with each request as part of the request body.