Home/ Questions/Q 3286928
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-17T20:26:09+00:00

I have an application that spiders websites for information. It seems like after 20-45

  • 0

I have an application that spiders websites for information. It seems like after 20-45 minutes of creating HttpWebRequests a bunch of them return timeouts. One thing we do is attach a BindIPDelegate anonymous function to give the request a specific IP since we round-robin through about 150 IPs.

I’m setting up the HttpWebRequest object with the following settings..

  • Setting User-Agent
  • Setting Keep-Alive to false so that the IP isn’t re-used
  • Setting TimeOut to 60000 (60 seconds)
  • Setting ReadWriteTimeout to 60000 (60 seconds)
  • Setting Proxy to null
  • Setting Accept to /
  • Setting CookieContainer to new CookieContainer
  • Setting Piplined to true
  • Setting Automatic Decompression to Deflate & GZIP

The application is using .NET 4.0 and running on Windows Server 2008 R2.

This definitely seems like something application/TCP/.NET related because if I restart the application it runs fine again. Also it appears more or less like the ones timing out are just queued up waiting on a local port or something.

Any ideas?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    You don’t say much about the code you actually use to perform the requests but, anyway, here are my guesses:

    1. You are using BeginGetResponse()/EndGetResponse() with a callback and the callback takes too long to complete (or blocks!). This could cause a deadlock in the threadpool if you are issuing a lot of requests in a short period of time.

    2. Since you are not reusing the connections and, again, if the requests happen very fast and non-stop, you might run out of sockets (last time I tried, ~3k per interface on windows). If setting KeepAlive to true fixes your problem, this is it.

    3. You are not calling Dispose()/Close() on the HttpWebRequest or the HttpWebResponse or the Stream you get from the response. This might work for a little bit until you hit the limit of 2 (from the MSDN docs) or 6 (configuration file default) in your application configuration settings for (system.net/connectionManagement/add[address=”*”,maxconnection=”6″]). A simple way to test if this is the problem is to set the limit to 1 and see if the problem happens earlier than before.

    Btw, setting KeepAlive to false and Pipelined to true does not make sense.

    • 0