Question

I have a java webapp using an ibatis row handler to load a very large dataset (1 million rows in an innodb table). The process is run as a nightly cron job by quartz scheduler. However, after it processes for 6 minutes, it dies with the following stack trace:

WARN [DefaultQuartzScheduler_Worker-8] MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(168) | Could not invoke method 'doBatch' on target object [org.myCron@4adb34]
org.springframework.jdbc.UncategorizedSQLException: SqlMapClient operation: encountered SQLException [  
--- The error occurred in org/myCron/mySqlMap.xml.  
--- The error occurred while applying a result map.  
--- Check the mySqlMap.outputMapping.  
--- The error happened while setting a property on the result object.  
--- Cause: com.mysql.jdbc.CommunicationsException: Communications link failure due to underlying exception: 

** BEGIN NESTED EXCEPTION ** 

java.io.EOFException

STACKTRACE:

java.io.EOFException
        at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:1903)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:2402)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:2860)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:771)
        at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1289)
        at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:362)
        at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:352)
        at com.mysql.jdbc.ResultSet.next(ResultSet.java:6106)
        at org.apache.commons.dbcp.DelegatingResultSet.next(DelegatingResultSet.java:168)
        at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:592)
        at com.ibatis.common.jdbc.logging.ResultSetLogProxy.invoke(ResultSetLogProxy.java:47)
        at $Proxy10.next(Unknown Source)
        at com.ibatis.sqlmap.engine.execution.SqlExecutor.handleResults(SqlExecutor.java:380)
        at com.ibatis.sqlmap.engine.execution.SqlExecutor.handleMultipleResults(SqlExecutor.java:301)
        at com.ibatis.sqlmap.engine.execution.SqlExecutor.executeQuery(SqlExecutor.java:190)
        at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.sqlExecuteQuery(GeneralStatement.java:205)
        at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithCallback(GeneralStatement.java:173)
        at com.ibatis.sqlmap.engine.mapping.statement.GeneralStatement.executeQueryWithRowHandler(GeneralStatement.java:133)
        at com.ibatis.sqlmap.engine.impl.SqlMapExecutorDelegate.queryWithRowHandler(SqlMapExecutorDelegate.java:649)
        at com.ibatis.sqlmap.engine.impl.SqlMapSessionImpl.queryWithRowHandler(SqlMapSessionImpl.java:156)
        at com.ibatis.sqlmap.engine.impl.SqlMapClientImpl.queryWithRowHandler(SqlMapClientImpl.java:133)
        at org.springframework.orm.ibatis.SqlMapClientTemplate$5.doInSqlMapClient(SqlMapClientTemplate.java:267)
        at org.springframework.orm.ibatis.SqlMapClientTemplate.execute(SqlMapClientTemplate.java:165)
        at org.springframework.orm.ibatis.SqlMapClientTemplate.queryWithRowHandler(SqlMapClientTemplate.java:265)
        at org.myCron.doBatch(MyCron.java:57)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:592)
        at org.springframework.util.MethodInvoker.invoke(MethodInvoker.java:248)
        at org.springframework.scheduling.quartz.MethodInvokingJobDetailFactoryBean$MethodInvokingJob.executeInternal(MethodInvokingJobDetailFactoryBean.java:165)
        at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:66)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:191)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:516)


** END NESTED EXCEPTION **

The stack trace is very vague. The only hints that I see are ‘the error happened while setting a property on the result object’. There are only two properties on the result object: a String and an Integer. Both of them permit null values, but my select statements indicate that neither of them have any null values. They both have a proper gettter/setter (which makes sense since the process runs for a while successfully before dying). Every time that the cron runs, it dies at a random point (so it isn’t stuck on a particular row).

Note – The method ‘doBatch’ does exist since that is the method that starts the cron process. If it couldn’t find doBatch, it couldn’t successfully process the first thousand rows.

I’ve also tried runnning the job outside of quartz and it also fails there as well. We tried increasing our MySQL net_read_timeout, net_write_timeout, and delayed_insert_timeout but none of these settings helped with the problem. I also tried setting my log4j setting to DEBUG and I did not get any helpful info.

Any other ideas about what I could try?

Answer 1

Sounds like MySQL closed the connection for some reason. Check the MySQL log see if anything shows up. Turn on various logging options for MySQL if necessary.

Also, start printing debug data (including timestamp) from your app – just print everything, then see what the last action was – perhaps you have some rarely triggered conditions in your code that has a bug.

I.e. every single time you talk to MySQL log it before AND after.