I have a fairly busy site which does around 10m views a month.
One of my app pools seemed to jam up for a few hours and I’m looking for some ideas on how to troubleshoot it..? I suspect that it somehow ran out of threads but I’m not sure how to determine this retroactively..? Here’s what I know:
- The site never went ‘down’, but around 90% of requests started timing out.
- I can see a high number of “HttpException – Request timed out.” in the log during the outage
- I can’t find any SQL errors or code errors that would have caused the timeouts.
- The timeouts seem to have been site wide on all pages.
- There was one page with a bug on it which would have caused errors on that specific page.
- The site had to be restarted.
The site is ASP.NET C# 3.5 WebForms..
Possibilities:
- Thread depletion: My thought is that the page causing the error may have somehow started jamming up the available threads?
- Global code error: Another possibility is that one of my static classes has an undiscovered bug in it somewhere. This is unlikely as the this has never happened before, and I can’t find any log errors for these classes, but it is a possibility.
UPDATE
I’ve managed to trace the issue now while it’s occurring. The pages are being loaded normally but for some reason WebResource.axd and ScriptResource.axd are both taking a minute to load. In the performance counters I can see ASP.NET Requests Queued spikes at this point.
The first thing I’d try is Sam Saffron’s CPU analyzer tool, which should give an indication if there is something common that is happening too much / too long. In part because it doesn’t involve any changes; just run it at the server.
After that, there are various other debugging tools available; we’ve found that some very ghetto approaches can be insanely effective at seeing where time is spent (of course, it’ll only work on the 10% of successful results).
You can of course just open the server profiling tools and drag in various .NET / IIS counters, which may help you spot some things.
Between these three options, you should be covered for: