Question

I do lose the connection to my database every time the API is idle for some time (the time cannot be defined precisely. It can happen after idle time of just some seconds, sometimes idle time after minutes, sometimes even right after jetty restart). This issue is reproducible on different machines, so I recon it has nothing to do with the mysql database directly.

The exception thrown can be found here

The database configuration is based upon hibernate and spring:

 <!-- c3p0 of version 0.9.2.1 (current stable) -->
 <!-- spring data: 1.2.0.RELEASE -->
 <!-- mysql connector: 5.1.18 -->
 <!-- hibernate: 4.1.9.FINAL --> 
 <bean id="basisDataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource">
    <property name="driverClass" value="${jdbc.driverClassName}" />
    <property name="jdbcUrl" value="${jdbc.url}" />
    <property name="user" value="${jdbc.username}" />
    <property name="password" value="${jdbc.password}" />
    <property name="minPoolSize" value="5" />
    <property name="maxPoolSize" value="20" />
    <property name="maxIdleTime" value="600" />
    <property name="maxConnectionAge" value="86400" />
</bean>

<bean id="entityManagerFactory"
    class="org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean">
    <property name="dataSource" ref="dataSource" />
    <property name="packagesToScan" value="de.yourdelivery.data.*" />
    <property name="jpaVendorAdapter">
        <bean class="org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter">
            <property name="generateDdl" value="false" />
            <property name="showSql" value="true" />
            <property name="database" value="MYSQL" />
        </bean>
    </property
</bean>

<bean id="transactionManager" class="org.springframework.orm.jpa.JpaTransactionManager">
    <property name="entityManagerFactory" ref="entityManagerFactory" />
</bean>

the Database connection and driver is a followed:

jdbc.driverClassName=com.mysql.jdbc.Driver
jdbc.url=jdbc:mysql://xxx:3306/xxx?zeroDateTimeBehavior=convertToNull

it does not even work with extending the jdbc.url by

&autoReconnect=true&autoReconnectForPools=true

I use OpenSessionInView Pattern to lazy load all my needed relations during dozer mapping (I know it is deemed to be an anti pattern, but it worked fine so far so currently not blaming it on that pattern)

Mysql wait_timeout is set to 28800

Startup log for C3PO looks like that

13:34:54,687 INFO PropertySourcesPlaceholderConfigurer:177 – Loading
properties file from file
[/opt/jetty/jetty-distribution-8.1.8.v20121106/webapps/rest/WEB-INF/classes/properties/customer.fidelity.points.properties]
13:34:54,687 INFO PropertySourcesPlaceholderConfigurer:177 – Loading
properties file from file
[/opt/jetty/jetty-distribution-8.1.8.v20121106/webapps/rest/WEB-INF/classes/properties/application.properties]
13:34:54,687 INFO PropertySourcesPlaceholderConfigurer:177 – Loading
properties file from file
[/opt/jetty/jetty-distribution-8.1.8.v20121106/webapps/rest/WEB-INF/classes/properties/restapi.properties]
13:34:54,688 INFO PropertySourcesPlaceholderConfigurer:177 – Loading
properties file from file
[/opt/jetty/jetty-distribution-8.1.8.v20121106/webapps/rest/WEB-INF/classes/properties/database.properties]
13:34:54,823 INFO MLog:80 – MLog clients using log4j logging.
13:34:54,883 INFO C3P0Registry:204 – Initializing c3p0-0.9.1.2 [built
21-May-2007 15:04:56; debug? true; trace: 10] 13:34:55,111 INFO
XmlWebApplicationContext:1350 – Bean ‘deDataSource’ of type [class
com.mchange.v2.c3p0.ComboPooledDataSource] is not eligible for getting
processed by all BeanPostProcessors (for example: not eligible for
auto-proxying) 13:34:55,148 INFO XmlWebApplicationContext:1350 – Bean
‘plDataSource’ of type [class
com.mchange.v2.c3p0.ComboPooledDataSource] is not eligible for getting
processed by all BeanPostProcessors (for example: not eligible for
auto-proxying) 13:34:55,150 INFO XmlWebApplicationContext:1350 – Bean
‘dataSource’ of type [class
de.yourdelivery.data.routing.DomainRouting] is not eligible for
getting processed by all BeanPostProcessors (for example: not eligible
for auto-proxying) 13:34:55,167 INFO XmlWebApplicationContext:1350 –
Bean
‘org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter#24bb6086’
of type [class
org.springframework.orm.jpa.vendor.HibernateJpaVendorAdapter] is not
eligible for getting processed by all BeanPostProcessors (for example:
not eligible for auto-proxying) 13:34:55,213 INFO
LocalContainerEntityManagerFactoryBean:264 – Building JPA container
EntityManagerFactory for persistence unit ‘default’ 13:34:55,332 INFO
Version:37 – HCANN000001: Hibernate Commons Annotations {4.0.1.Final}
13:34:55,338 INFO Version:41 – HHH000412: Hibernate Core
{4.1.9.Final} 13:34:55,341 INFO Environment:239 – HHH000206:
hibernate.properties not found 13:34:55,342 INFO Environment:342 –
HHH000021: Bytecode provider name : javassist 13:34:55,361 INFO
Ejb3Configuration:527 – HHH000204: Processing PersistenceUnitInfo [
name: default
…] 13:34:55,795 INFO ConnectionProviderInitiator:188 – HHH000130: Instantiating explicit connection provider:
org.hibernate.ejb.connection.InjectedDataSourceConnectionProvider
13:34:55,880 INFO AbstractPoolBackedDataSource:462 – Initializing
c3p0 pool… com.mchange.v2.c3p0.ComboPooledDataSource [
acquireIncrement -> 3, acquireRetryAttempts -> 30, acquireRetryDelay
-> 1000, autoCommitOnClose -> false, automaticTestTable -> null, breakAfterAcquireFailure -> false, checkoutTimeout -> 0,
connectionCustomizerClassName -> null, connectionTesterClassName ->
com.mchange.v2.c3p0.impl.DefaultConnectionTester, dataSourceName ->
1hge17b8s1e4h6uz10tbcor|3efe0ce9, debugUnreturnedConnectionStackTraces
-> false, description -> null, driverClass -> com.mysql.jdbc.Driver, factoryClassLocation -> null, forceIgnoreUnresolvedTransactions ->
false, identityToken -> 1hge17b8s1e4h6uz10tbcor|3efe0ce9,
idleConnectionTestPeriod -> 0, initialPoolSize -> 3, jdbcUrl ->
jdbc:mysql://192.168.1.10:3306/xxxx?zeroDateTimeBehavior=convertToNull, maxAdministrativeTaskTime -> 0, maxConnectionAge -> 86400, maxIdleTime
-> 600, maxIdleTimeExcessConnections -> 0, maxPoolSize -> 20, maxStatements -> 0, maxStatementsPerConnection -> 0, minPoolSize -> 5,
numHelperThreads -> 3, numThreadsAwaitingCheckoutDefaultUser -> 0,
preferredTestQuery -> null, properties -> {user=*,
password=*}, propertyCycle -> 0, testConnectionOnCheckin ->
false, testConnectionOnCheckout -> false, unreturnedConnectionTimeout
-> 0, usesTraditionalReflectiveProxies -> false ] 13:34:56,086 INFO Dialect:123 – HHH000400: Using dialect:
org.hibernate.dialect.MySQLDialect 13:34:56,095 INFO
LobCreatorBuilder:120 – HHH000424: Disabling contextual LOB creation
as createClob() method threw error :
java.lang.reflect.InvocationTargetException 13:34:56,108 INFO
TransactionFactoryInitiator:73 – HHH000268: Transaction strategy:
org.hibernate.engine.transaction.internal.jdbc.JdbcTransactionFactory
13:34:56,112 INFO ASTQueryTranslatorFactory:48 – HHH000397: Using
ASTQueryTranslatorFactory 13:34:56,149 INFO Version:24 – Hibernate
Validator 4.2.0.Final 13:34:56,916 INFO XmlWebApplicationContext:1350
– Bean ‘entityManagerFactory’ of type [class org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean] is
not eligible for getting processed by all BeanPostProcessors (for
example: not eligible for auto-proxying)

So why is the application losing its connection during idle time?

Answer 1

Interesting problem

It can happen due to possibly multiple reasons. your app server is losing connections to your DB. It is hard to tell looking at the logs that only connections which are idle are the only one which are getting killed.
Your DB server might be not allocating connections. I hope you have checked your DB logs for errors. How you are handling transactions in your services ? Are you using spring transactions ?

It’s a good idea always to specify destroy-method if you are using external connection pool provider , so as to shutdown the data source immediately when the spring application shuts down

<bean id="basisDataSource" class="com.mchange.v2.c3p0.ComboPooledDataSource" destroy-method="close">

Change the following in your c3p0 pool settings

Take two approaches
Remove maxConnectionAge & maxIdleTime

Test by changing below two parameters , they are expensive in terms of performance hit

testConnectionOnCheckout =true 
preferredTestQuery =SELECT 1

In case if you are able to resolve it using the above settings

Change the above settings to below as the below is less expensive in terms of performance hit

idleConnectionTestPeriod – 30 
preferredTestQuery =SELECT 1

Post your findings