I’ve got a problem with our live server. Occasionally a page will not load, and the following error is displayed in plain black text on a white background:
Server Error
The server encountered an internal error and was unable to complete your request
Application server is busy. Either there are too many concurrent requests
If I look in the error log for the website, at the time of the error using the following command:
grep "12:17:33" error_log
lines such as this are returned:
[Tue May 03 12:17:33 2011] [notice] jrApache[14352: 21512] returning error page for JRun too busy or out of memory
Then if I search in the Coldfusion cfserver.log I find these errors, a few seconds before hand:
05/03 12:17:29 error Corrupt form data: premature ending
java.io.IOException: Corrupt form data: premature ending
at com.oreilly.servlet.multipart.MultipartParser.(MultipartParser.java:177)
at com.oreilly.servlet.multipart.MultipartParser.(MultipartParser.java:99)
at coldfusion.filter.FormScope.fillMultipart(FormScope.java:190)
at coldfusion.filter.FusionContext.SymTab_initForRequest(FusionContext.java:369)
at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:33)
at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:53)
at coldfusion.filter.RequestThrottleFilter.invoke(RequestThrottleFilter.java:126)
at coldfusion.CfmServlet.service(CfmServlet.java:200)
at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
at jrun.servlet.FilterChain.service(FilterChain.java:101)
at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)
05/03 12:17:29 error (JRun Service: ProxyService [jrun.servlet.jrpp.JRunProxyService@500b675]) JRunPRoxyServer.invokeRunnable:
java.lang.IllegalStateException
at jrun.servlet.JRunResponse.getWriter(JRunResponse.java:205)
at jrun.servlet.JRunResponse.sendError(JRunResponse.java:597)
at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:328)
at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)
java.lang.IllegalStateException
at jrun.servlet.JRunResponse.getWriter(JRunResponse.java:205)
at jrun.servlet.JRunResponse.sendError(JRunResponse.java:597)
at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:328)
at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)
I don’t really know what to do from here.
If the page is refreshed then it loads without error. I don’t know how to replicate the error, it doesn’t seem to happen on any particular page or at any articular time.
I’ve tried setting:
Maximum number of simultaneous Template requests: 100
in the Coldfusion administrator, I’ve done that based on Steven Erat’s post: http://www.talkingtree.com/blog/index.cfm/2005/3/11/ContextSwitchingBad
But I’m still getting errors. Would increasing this to a higher value help?
And then there’s the “premature form ending” part of the longer Java error, I’m not sure if that’s related and don’t understand what it means anyway.
I’ve also looked through my access logs, and ruling out any images or js files that get called at the time of any errors, I’ve got a variety of CFM pages that return a 503 http status, and some of them are CFM pages that get returned with a content type of Javascript, and are never requested after a form post.
So in summary, I’m getting some pretty blunt error messages, and I can reproduce them at present, and the page loads fine if I just hit F5, and I’d like to know if anyone can help me to understand the source of the problem.
Oh and one last thing, we’re running a RHEL 5 server with CF9, and we’ve upgraded recently from CF7. Before the upgrade this error was not happening.
I’m really sorry to answer this myself, and I’m not entirely sure what fixed it.
But I installed FusionReactor, which required me updating my JVM, and then rebooting coldfusion. Since then, the problem has not occurred.
I was getting it daily, at least hourly but at peak times a couple of times a minute. Now it hasn’t happened in 5 days.
I’d love to hear any thoughts on why any of these might have fixed it?