Home/ Questions/Q 7906521
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-03T11:00:53+00:00

My problem is this: I am trying to process about 1.5 million rows of

  • 0

My problem is this: I am trying to process about 1.5 million rows of data in Spring via JDBCTemplate coming from MySQL. With such a large number of rows, I am using the RowCallbackHandler class as suggested here

The code is actually working, but’s SLOW… The thing is that no matter what I set the fetch size to, I seem to get approximately 350 records per fetch, with a 2 to 3 second delay between fetches (from observing my logs). I tried commenting out the store command and confirmed that behavior stayed the same, so the problem is not with the writes.

There are 6 columns, only 1 that is a varchar, and that one is only 25 characters long, so I can’t see throughput being the issue.

Ideally I’d like to get more like 30000-50000 rows at a time. Is there a way to do that?

Here is my code:


    protected void runCallback(String query, Map params, int fetchSize, RowCallbackHandler rch) 
            throws DatabaseException {
        int oldFetchSize = getJdbcTemplate().getFetchSize();
        if (fetchSize > 0) {
            getJdbcTemplate().setFetchSize(fetchSize);
        }
        try {
            getJdbcTemplate().query(getSql(query), rch);
        }
        catch (DataAccessException ex) {
            logger.error(ExceptionUtils.getStackTrace(ex));
            throw new DatabaseException( ex.getMessage() );         
        }
        getJdbcTemplate().setFetchSize(oldFetchSize);
    }

and the handler:

public class SaveUserFolderStatesCallback implements RowCallbackHandler {
        @Override
        public void processRow(ResultSet rs) throws SQLException {
            //Save each row sequentially.
            //Do NOT call ResultSet.next() !!!!

            Calendar asOf = Calendar.getInstance();
            log.info("AS OF DATE: " + asOf.getTime());
            Long x = (Long) rs.getLong("x");
            Long xx = (Long) rs.getLong("xx");
            String xxx = (String) rs.getString("xxx");
            BigDecimal xxxx = (BigDecimal)rs.getBigDecimal("xxxx");
            Double xxxx = (budgetAmountBD == null) ? 0.0 : budgetAmountBD.doubleValue();
            BigDecimal xxxxx = (BigDecimal)rs.getBigDecimal("xxxxx");
            Double xxxxx = (actualAmountBD == null) ? 0.0 : actualAmountBD.doubleValue();           

            dbstore(x, xx, xxx, xxxx, xxxxx, asOf);
        }       

    }
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    The answer actually is to do setFetchSize(Integer.MIN_VALUE) while this totally violates the stated contract of Statement.setFetchSize, the mysql java connector uses this value to stream the resultset. This results in tremendous performance improvement.

    Another part of the fix is that I also needed to create my own subclass of (Spring) JdbcTemplate that would accomodate the negative fetch size… Actually, I took the code example here, where I first found the idea of setting fetchSize(Integer.MIN_VALUE)

    http://javasplitter.blogspot.com/2009/10/pimp-ma-jdbc-resultset.html

    Thank you both for your help!

    • 0