.net 2.0 aspx app / IIS6 creating a silly number of threads in w3wp.exe process app pool.
The app has been isolated to its own app pool with the following settings:
RECYCLING
recycle worker processses (in minutes) : 870
recycle worker process (no of requests): (not ticked)
recycle worker processes at the following times: 00:00
max virtual memory: (not ticked)
max used memory (in mb): 1000mb (1gb)
PERFORMANCE
shutdown worker processes after being idle for (time in mins): 20
limit the kernal request queue (number of requests): 1000
enable cpu monitoring (%): 85
refresh cpu usage numbers (in mins): 5
action performed when cpu usage exceeds maximum cpu uses: NO ACTION (keeps
sessions)
max number of worker processes: 1
HEALTH
enable pinging (checked)
ping worker process every (seconds) : 30
enable rapid fail protection (checked)
failures: 5
time period (in mins):5
start time limit – worker process must startup within (seconds): 90
shutdown time limit – worker process must shutdown within (seconds): 90
Normal running would see the w3wp.exe process utilise 300MB ram and 50
therads. When my problem occurs the thread count slowly increases to 10,000
, ram to 1GB before the threads are knocked back to 0. The w3wp.exe process
is NOT shutdown and my users are not logged out (crucially), ie they keep
their session and dont have to log back in . Altough the standard 50 threads
are killed in amongst the 10, 000 rouge threads.
1) Can an expert offer any pros/cons on the above app pool settings ?
2) The “max used mem” setting appears to be doing the trick to
automatiaclly handle this issue (by killing the threads, keeping the session
alive , but can someone explain why ? … i take it threads are unrelated to
the session).
The app uses server based sessions but we store a local cookie for authentication.
Threads
10k threads is insanely high, and your threads are spending more time hopping on and off the processor than doing actual work. aka thrashing.
EDIT: I’m assuming here that it’s a .NET web application.
Does your application use a ThreadPool or BackgroundWorkers? It seems like you’d have to be using some mechanism other than IIS’s standard thread entourage (which is only around 4 per processor) to reach 10k threads.
Memory
Each thread requires memory to keep track of in addition to the memory it utilizes for work, so by sheer volume of threads, you are probably reaching the 1G limit.
Session (I will survive!)
The application is probably setup to store Session State in persistent storage or the session state service. With this being the case, the worker process can safely be recycled without losing user state information. If the session state was configured (in the Web.config) as In-Proc, then session state would be lost when the worker process recycled.
Work process recycling
One other thing of note, before a worker process dies, another worker process is setup and started to take it’s place. It’s somewhere in this process that you’re probably seeing the w3wp.exe process (either old or new) with 0 threads.
BackgroundWorkers are like rabbits
If your threads are performing work that lasts longer than 1 second (1/2 second really), don’t use BackgroundWorkers. Unless you change the ThreadPool’s max threads (which is NOT recommended, as this can screw up deeper functionality within .NET), there’s not a hard (enough) limit on the number of BackgroundWorkers that can run concurrently. It would be better to use a Producer Consumer Queue model in that case.
Check out this site. It’s an awesome resource on concurrent programming with lots of models and examples.