Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
I’m trying to scrape a quote engine but HttpWebResponse returns only the “please wait” screen, not the final result.
Is there a way to make the request wait for the response? Status codes won’t work because the “please wait” screen returns 200. – I guess it redirects after it retrieved the results from the database but that redirect comes after the initial 200.
AllowAutoRedirect is true but it doesn’t make any difference in this case.
Has any of you seen this before?
The site is probably using either Javascript or a meta refresh tag to load the ultimate destination page. Neither of those are directly supported by HttpWebRequest. You’ve got the source to the “please wait” screen – read it to see exactly how they are redirecting your browser to the end result. There may be a URL you can parse out of the content for the “please wait” screen. Once you have that URL, you can just load it.