Question

My app pools keep randomly crashing in IIS 6.0 MS Debug Diag points to kernel32.dll every time.

The entry point is always mscorwks!CreateApplicationContext+bbef and the result is always a System.UnauthorizedAccessException.

Stack Trace:

Function                                       Arg 1        Arg 2        Arg 3   
kernel32!RaiseException+3c                     e0434f4d     00000001     00000001    
mscorwks!GetMetaDataInternalInterface+84a9     18316b3c     00000000     00000000    
mscorwks!GetAddrOfContractShutoffFlag+ac01     18316b3c     00000000     023cfbd8    
mscorwks!GetAddrOfContractShutoffFlag+ac73     00000000     000e8c88     8038b2d0    
mscorwks!GetAddrOfContractShutoffFlag+aca4     18316b3c     00000000     023cfbe4    
mscorwks!GetAddrOfContractShutoffFlag+acb2     18316b3c     acc05c33     7a399bf0    
mscorwks!CoUninitializeCor+67be                00000000     023cfc1c     023cfc8c    
mscorwks!CoUninitializeCor+87a1                001056e8     79fd87f6     023cfeb0    
mscorwks!CorExitProcess+4ad3                   023cfeb0     023cfd20     79f40574    
mscorwks!CorExitProcess+4abf                   001056e8     79f405a6     023cfd04    
mscorwks!CorExitProcess+4b3e                   000e8c88     00000000     023cfda7    
mscorwks!StrongNameErrorInfo+1ddab             00000000     00000000     023cfeb0    
mscorwks!StrongNameErrorInfo+1e07c             023cfeb0     00000000     00000000    
mscorwks!CoUninitializeEE+4e0b                 023cfeb0     023cfe5c     79f7762b    
mscorwks!CoUninitializeEE+4da7                 023cfeb0     acc05973     00000000    
mscorwks!CoUninitializeEE+4ccd                 023cfeb0     00000000     001056e8    
mscorwks!GetPrivateContextsPerfCounters+f1cd   79fc24f9     00000008     023cff14    
mscorwks!GetPrivateContextsPerfCounters+f1de   79fc24f9     acc058c3     00000000    
mscorwks!CorExeMain+1374                       00000000     00000003     00000002    
mscorwks!CreateApplicationContext+bc35         000e9458     00000000     00000000    
kernel32!GetModuleHandleA+df                   79f9205f     000e9458     00000000   

Does anybody know what this means and how to fix it?

Edit: The above stack trace turned out to be a symptom and not the cause. The above stack trace only shows the unmanaged stack but the problem happened in managed code. I used the steps in my answer below to dig into the crash dump and extract the managed exception.

Answer 1

The reference to mscorwks.dll was only a symptom of the problem. mscorwks.dll is the dll that contains the common language runtime.

To diagnose the root cause of the problem, I followed the following steps:

1. Use DebugDiag to capture a crash dump when IIS recycled the app pool.
2. Open the crash dump in windbg.
3. Type ".loadby sos mscorwks" (no quotes) at the windbg command line to load the CLR debug tools
4. Use the !PrintException command to print out the last exception recorded in the crash dump. This is most likely the one that caused the IIS app pool recycle
5. Use the !clrstack command to view the stack on the current thread that threw the exception
6. More command references for windbg can be found here and here. Lastly, this MSDN blog has great tutorials on using windbg.

Good luck on your debuging adventure!