Question

Here’s the purpose of my console program: Make a web request > Save results from web request > Use QueryString to get next page from web request > Save those results > Use QueryString to get next page from web request, etc.

So here’s some pseudocode for how I set the code up.

 for (int i = 0; i < 3; i++)         {             strPageNo = Convert.ToString(i);                //creates the url I want, with incrementing pages             strURL = 'http://www.website.com/results.aspx?page=' + strPageNo;                 //makes the web request             wrGETURL = WebRequest.Create(strURL);              //gets the web page for me             objStream = wrGETURL.GetResponse().GetResponseStream();              //for reading web page             objReader = new StreamReader(objStream);              //--------             // -snip- code that saves it to file, etc.             //--------              objStream.Close();             objReader.Close();              //so the server doesn't get hammered             System.Threading.Thread.Sleep(1000);           } 

Pretty simple, right? The problem is, even though it increments the page number to get a different web page, I’m getting the exact same results page each time the loop runs.

i IS incrementing correctly, and I can cut/paste the url strURL creates into a web browser and it works just fine.

I can manually type in &page=1, &page=2, &page=3, and it’ll return the correct pages. Somehow putting the increment in there screws it up.

Does it have anything to do with sessions, or what? I make sure I close both the stream and the reader before it loops again…

Answer 1

Have you tried creating a new WebRequest object for each time during the loop, it could be the Create() method isn’t adequately flushing out all of its old data.

Another thing to check is that the ResponseStream is adequately flushed out before the next loop iteration.