The setup: 2 web servers and a seperate state server
I have two production web servers in a load balanced configuration. The ASP.NET web app they host shares state (like a web farm) using this line in their web.configs:
<sessionState mode="StateServer" stateConnectionString="tcpip=9.9.9.9:42424" cookieless="false" timeout="60"/>
9.9.9.9 is the IP of the machine the asp.net session state service is running on (ok it’s not 9.9.9.9 really, changed to protect the innocent). It’s a third machine (the database server, actually.
It worked fine until…
The error: website down!
Suddenly the site went down, just showing a generic asp.net error page (‘turn custom errors off to see this error’ or whatever).
The app’s log recorded the actual error message:
An unhandled exception occurred Unable to make the session state request to the session state server. Please ensure that the ASP.NET State service is started and that the client and server ports are the same. If the server is on a remote machine, please ensure that it accepts remote requests by checking the value of HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\aspnet_state\Parameters\AllowRemoteConnection. If the server is on the local machine, and if the before mentioned registry value does not exist or is set to 0, then the state server connection string must use either ‘localhost’ or ‘127.0.0.1’ as the server name.
So it appears that the web app was unable to contact the state server (9.9.9.9).
I “tried turning if auf and then onnegen” – restarting the state server fixed the problem.
Why?
I really want know what happened and why so I can prevent it happening again.
So far all I have are two theories:
-
A windows update, to .net framework 4, was applied around that time on the state server. So maybe the update did something to the asp.net state service? The windows event viewer showed that .net 4 had logged a warning around then:
Updates to the IIS metabase were aborted because IIS is either not installed or is disabled on this machine. To configure ASP.NET to run in IIS, please install or enable IIS and re-register ASP.NET using aspnet_regiis.exe /i.
-
Some kind of temporary network problem between the prod web sites and the state server? They do sit right next to each other in the same physical rack though.
-
??? Any other ideas, anyone?
Anyone seen this before, or able to correct me on anything?
Has this happened since? The easy answer is that the problem was with the db server, not the web app. Are there any relevant errors in the log on the db server?
The fact that both apps threw an error indicates that a common resource was the problem. We chased a similar issue for a good solid week awhile back, and eventually found a faulty fiber channel gadget. (that’s below my OSI level, not sure about the details).