This is kind of weird problem. I have been using HttpWebRequest for a long time now. But never faced this problem before. The site I am scrapping is huge in size, atleast 3mb each page.
In xp it is giving no error but
scrapping incomplete page.
In win7 or 2008 server it is showing this error
“Unable to read data from the
transport connection: An existing
connection was forcibly closed by the
remote host”
. Any help will be very much appreciated.
After spending 5 days on it I come to this conclusion that this is a big bug of .net
And atlast I solved this problem using WebBrowser component. Though I dont like it very much because it does not work outside of main thread. But it is really fast and scrapping those pages like a champ.