We’ve been experiencing a very serious issue with IIS and Azure. Not sure if this is something on IIS side or our custom code side of things.
I’m involved with two websites that are running in Azure (Site A and Site B). (Standard WebRoles, ASP.NET MVC3). Both of theses sites are architected completely differently and have nothing to do with one another, but both have shown similar behavior under similar circumstances.
Site A is running with Session state enabled. Session is stored in SQL Azure database. Most of the calls to Site A are secured via ASP.NET SQL Membership provider that points to SQL Azure database
Site B is running with Session state enabled as well. Session is stored in Azure AppFabric cache. Site B also has an http handler that talks to AppFabric Cache and Azure Table Storage.
Issues begin but do NOT recover when key Azure resources (like SQL Azure or Cache) become very slow. When these resources become very slow and processing time for each request exceeds a minute, Azure’s load balancer terminates those connections but IIS on the web roles is not clearing/removing these requests from its active queue.
So, the issue is not that the sites are non-responsive when SQL Azure or AppFabric Cache is very slow. The huge issue is that the sites do not recover when SQL Azure or AppFabric Cache come back and start working normally. Requests are sitting in the Active Requests list and do not go away for long time (hours?). Frankly, I don’t know how long they sit there for, because we reboot those servers ASAP. Since Azure resources can have occasional intermittent issues, and since traffic to both sites is very high, both sites quickly boggle down under the weight of non-clearing requests. IIS queue fills up and until someone goes in and restarts the App Pools, the sites are unavailable.
I had a similar problem, IIS requests that would last for a very long time. I spent a long time trying to figure out why IIS wasn’t killing them. I tried Sandrino Di Mattia’s solution, and others, none of which worked for me.
It turned out that IIS wasn’t killing the requests because they were still active. In some cases the client browsers would open a connection and just hold it forever. I’d look in the network debugger for the browser (Firebug, the Webkit inspector, etc) and see the requests just sitting there spinning. As far as I can tell they were responding to keepalives, so IIS and the load balancers would keep the connection and request active. The ultimate solution was to make the browsers just not do that.
It may not be related to your problem, but in my particular case the problem was
<video>tags. When given a<video>tag pointed to a large file, they would open a connection immediately and hold it until the video was played, which might be never (we had autoplay turned off). The solution was to not create the<video>tags until we were ready to play the video.Also, is there some way to know that the Azure load balancer has killed a request? The only way I know of to see active is requests is via the IIS management console.