Question

I am running a web application in Production that recently crashed after it was under some stress. I would guess 100-300 people were accessing the site at similar times, which I would expect to work fine.

The logs around the time of the crash are:

org.codehaus.groovy.grails.web.taglib.exceptions.GrailsTagException: Error executing tag <g:render>: Hibernate operation: could not inspect JDBC autocommit mode; uncategorize\
d SQLException for SQL [???]; SQL state [null]; error code [0]; Cannot get a connection, pool error Timeout waiting for idle object; nested exception is org.apache.commons.db\
cp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object at /WEB-INF/grails-app/views/layouts/file.gsp:37
        at gsp_file_gsp$_run_closure2.doCall(file_gsp.groovy:43)
        at gsp_file_gsp$_run_closure2.doCall(file_gsp.groovy)
        at gsp_file_gsp.run(gsp_file_gsp.groovy:48)
        at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:190)
        at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:291)
        at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:774)
        at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:703)
        at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:896)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.springframework.jdbc.UncategorizedSQLException: Hibernate operation: could not inspect JDBC autocommit mode; uncategorized SQLException for SQL [???]; SQL stat\
e [null]; error code [0]; Cannot get a connection, pool error Timeout waiting for idle object; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot get a co\
nnection, pool error Timeout waiting for idle object
        at User.find(User.groovy:68)
        at User$find.call(Unknown Source)
        at gsp_pps_file_gsp.run(gsp_file_gsp.groovy:22)
        ... 9 more
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool error Timeout waiting for idle object
        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:114)
        at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:1044)
        at $Proxy7.getAutoCommit(Unknown Source)
        ... 12 more
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
        at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1144)
        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:106)
        ... 14 more

I believe this is directly related to some code I added recently to my Grails project that affect the Connection Pool (although I believe this is not a grails specific problem):

maxActive = 50
maxIdle = 15
minIdle = 5
initialSize = 15
minEvictableIdleTimeMillis = 180000
timeBetweenEvictionRunsMillis = 180000
maxWait = 10000
validationQuery = "/* ping */"

What am I doing wrong? Please help! Thank you.

Answer 1

I had bad code that was keeping connections alive way longer than they should be. Once I fixed the underlying problem this went away.