We have an IIS hosted web method which is randomly dying on us about 10% of the time. In trying to debug this we’ve added Log.Debug() messages in front of every real code line and it appears to be dying on random lines.
Has anyone seen this or have an idea on how to debug this?
[Additional Details]
We’ve spent a lot of time looking at it and have discovered the following…
-
We have a seperate self-hosted WCF Service that access the same database and lives on the same machine. When it is under heavy load the web method croaks every time. If it’s not under load then things usually work fine (but not 100%).
-
High CPU doesn’t seem to be part of the problem. We ran a small app that created a high cpu load and the web service did not die.
-
The web service dies when we either new up an XmlSerializer (without doing the sgen precomp) OR have NHibernate create a SessionFactory. The only two things these things have in common is that they 1) seem like things people commonly do.. 2) seem like they would be fairly intensive.
-
We’ve added a Global.asax to try to capture Application_End and Application_Error but neither event gets fired. This to me implies that we’re not dealing with a normal application pool resetting?
Figured it out.. two problems really..
We added Global.asax but it didn’t get copied over which explains why we weren’t seeing any messages. We fixed this and found out that…
Our WCF log was being written out to the bin directory of the IIS Web Service. In retrospect this is kind of silly since the WS is an old school web service. The WCF stuff is in the same directory only for some reason that is unknown to us since the initial person who set things up is gone..
Lesson learned.. Somewhere there is a message that explains everything.. you just have to find it.