Question

I am using the groovy.sql.Sql class to query a database and process the results. My problem is that the ResultSet can be very large; so large that I risk running out of memory if I try to process the whole ResultSet at once. I know the Sql.rows() method supports paging using offset and max results parameters but I haven’t been able to find a good example of how to use it (and I’m not certain that paging is what I’m looking for).

Basically, here’s what I’m trying to do:

def endOfResultSet = false
for(int x = 1; !endOfResultSet; x+=1000){
    def result = sql.rows("Select * from table", x, 1000)
    processResult(result)
    endOfResultSet = result.size()!=1000
}

My question is if Groovy is smart enough to reuse the same result set for the sql.rows("Select * from table", x, 1000) call or if it will be repeatedly be running the same statement on the database and then paging to where the offset starts.

Your help is appreciated, Thanks!

Edit: What I’m trying to avoid is running the same query on the database multiple times. I’d like to run the query once, get the first 1,000 rows, process them, get the next 1,000 rows, etc… until all the rows are processed.

Answer 1

I assume you’ve seen this blog post about paging?

To answer your question, if we look at the code for the Sql class in Groovy, we can see that the code for rows(String,int,int) calls rows(String,int,int,null)

And the code for that is:

    AbstractQueryCommand command = createQueryCommand(sql);
    ResultSet rs = null;
    try {
        rs = command.execute();
        List<GroovyRowResult> result = asList(sql, rs, offset, maxRows, metaClosure);
        rs = null;
        return result;
    } finally {
        command.closeResources(rs);
    }

So as you can see, it gets the full ResultSet, then steps through this inside the asList method, filling a List<GroovyRowResult> object with just the results you requested.

Edit (after the question was edited)

As I said in my comment below, I think you’re going to need to write your own paging query for the specific database you are using… For example, with MySQL, your above query can be changed to:

def result = sql.rows( "SELECT * FROM table LIMIT ${Sql.expand x}, 1000" )

Other databases will have different methods for this sort of thing…I don’t believe there is a standard implementation