Question

We have jobs that might process up to 20,000 files. We are using a MultiResourcePartitioner to set things up. The job does run, but we have noticed a bottleneck.

SpringBatch is creating entries in the BATCH_STEP_EXECUTION table for each file found, and will not process any files until it has created a table entry for every file. The loading of this table seems to take a very long time.

In local testing, trying to process just 1,000 files, it is taking 38-40 minutes to add the rows to ‘BATCH_STEP_EXECUTION’. Once the table is loaded, the files are processed quite rapidly (usually under 1 minute).

I would hope that this is not typical behavior and that I am just missing something.

Here is how the database is set up (we really subclass the ‘OracleDataSource’ (we are using ‘ojdbc6.jar’ file to get to the class) and the db_file is a properties file to get to the url, password, etc.):

<bean id="dataSource" class="oracle.jdbc.pool.OracleDataSource" destroy-method="close">
    <constructor-arg value="db_file" /> 
       <property name="connectionCachingEnabled" value="true" />  
       <property name="connectionCacheProperties"> 
         <props merge="default">   
            <prop key="InitialLimit">10</prop>       
            <prop key="MinLimit">25</prop>           
            <prop key="MaxLimit">50</prop>     
            <prop key="InactivityTimeout">1800</prop>
            <prop key="AbandonedConnectionTimeout">900</prop>
            <prop key="MaxStatementsLimit">20</prop> 
            <prop key="PropertyCheckInterval">20</prop>     
         </props>     
       </property>
</bean> 

Here is the rest of the JobRepository definition:

<bean id="transactionManager" class="org.springframework.jdbc.datasource.DataSourceTransactionManager">
    <property name="dataSource" ref="dataSource" />
</bean> 


<bean id="jobRepository" class="org.springframework.batch.core.repository.support.JobRepositoryFactoryBean" >
    <property name="databaseType" value="oracle" />
    <property name="dataSource" ref="dataSource" />
    <property name="transactionManager" ref="transactionManager" />
    <property name="isolationLevelForCreate" value="ISOLATION_DEFAULT"/>
</bean> 

<bean id="jobExplorer" class="org.springframework.batch.core.explore.support.JobExplorerFactoryBean">
    <property name="dataSource" ref="dataSource" />
</bean> 

<bean id="jobLauncher" class="org.springframework.batch.core.launch.support.SimpleJobLauncher"> 
    <property name="jobRepository" ref="jobRepository" /> 
</bean>

<bean id="jobParametersIncrementer" class="org.springframework.batch.core.launch.support.RunIdIncrementer" />   

Anyone have any ideas?

Answer 1

As an FYI, SpringSource has identified this as a bug: Batch-1908.

As a workaround, we are simply lowering the number of files to process with a given run, and then increasing the number of times that the job runs in a given day.

We are using 2,000 as our file limit as it provides acceptable performance.