Question

I have an excel sheet with about 25,000 rows. Each row in the excel sheet will be a row in my table as well. I tried to do the following and it just keeps me giving Memory out of bound exception. I tried to change the batchSize from 25 to 50, 100, 500. None of them works. Can anyone tell me what am I doing wrong? changing the heap size of the JVM is not an option for me.

public void saveForecast(List list) throws FinderException{
    final Session session = getCurrentSession();
    final int batchSize = 25;
    Connection con = null;
    PreparedStatement pstmt = null;
    Iterator iterator = list.iterator();
    int rowCount = list.size();
    String sqlStatement = "INSERT INTO DMD_VOL_UPLOAD (ORIGIN, DESTINATION, DAY_OF_WEEK, EFFECTIVE_DATE, DISCONTINUE_DATE, VOLUME)";
    sqlStatement += " VALUES(?, ?, ?, ?, ?, ?)";
    System.out.println(sqlStatement);
    System.out.println("Number of rows to be inserted: "+ rowCount);
    System.out.println("Starting time: "+new Date().toString());
    try{
        con = session.connection();
        for(int i=0; i<rowCount; i++){
            ForecastBatch forecastBatch = (ForecastBatch) iterator.next();              
            pstmt = con.prepareStatement(sqlStatement);             
            pstmt.setString(1, forecastBatch.getOrigin());
            pstmt.setString(2, forecastBatch.getDestination());
            pstmt.setInt(3, forecastBatch.getDayOfWeek());

            java.util.Date effJavaDate = forecastBatch.getEffectiveDate();
            java.sql.Date effSqlDate = new java.sql.Date(effJavaDate.getTime());                
            pstmt.setDate(4,  effSqlDate);              
            java.util.Date disJavaDate=forecastBatch.getDiscontinueDate();
            java.sql.Date disSqlDate =  new java.sql.Date(disJavaDate.getTime());   

            pstmt.setDate(5, disSqlDate);               
            pstmt.setInt(6, forecastBatch.getVolumeSum());

            pstmt.addBatch();
            if(i % batchSize == 0){
                pstmt.executeBatch();
                session.flush();
                session.clear();
            }
        }
        pstmt.executeBatch();
        pstmt.close();
        System.out.println("Ending Time: "+ new Date().toString());
    }catch(SQLException e){
        e.printStackTrace();
        throw new FinderException(e);
    }
    finally{
        HibernateUtil.closeSession();
    }
}

}

Answer 1

You are creating a new statement inside your loop but only closing the last statement after the loop ends. That means you’re actually creating 25000 statements and closing only a single one leaving 24999 statements open, which I’m not surprised is causing you to run out of resources.

Furthermore, you’re not using the batch statements correctly (you’d have to create the statement once, then set the parameters, call addBatch, set more parameters, call addBatch again, and so on, then call executeBatch when you want to submit all values in the batch.

EDIT:

You’ll probably fix this by moving the prepareStatement call just before the for loop and I don’t think calling session flush/clear is necessary either.