Question

I have a method to get a load of objects from a database, which returns an Iterable.

For now, I am loading a resultset from a database, building objects from it and populating a collection with those objects.

Obviously, I’m memory constrained as to how much data can be loaded using this method and if I run out Bad Things Happen.

I’d like to modify the implementation to chunk data from the database instead of getting it all at once and then expose the resulting objects to the client through the Iterable interface. My database drivers can do their bit, so my first thought is a custom implementation of Iterable that does this.

Is this a good approach? It strikes me as something that might already be supported in the runtime or libraries – not involving ORM solutions please.

Answer 1

Personally the simplest solution that I can think of is to implement a Iterator as a thin wrapper around a ResultSet. That has several advantages:

• You don’t need to provide reproducable SQL statements (you can stream non-sorted results, for example)
• You don’t need to rely on repeatable read, which can be costly
• If your JDBC driver is good, then you can just use its streaming result features (warning: some JDBC drivers always grab the full result as soon as you start iterating over it!)
• You don’t need to implement re-starting the Iterator (Iterable.iterator() could be called twice, which makes this complicated).
• Not “remembering” previously returned data means that the memory requirement can be held pretty low

It also has a few disadvantages:

• your Iterator implementation effectively becomes an external resources, since it binds a JDBC resource: it must be “closed” in some way, making it harder to use
• if the Iterator hangs around for a longer time, then that also lets a JDBC Connection hang around, which might be needed elsewhere (you can’t return it to the pool, until the Iterator is done).

An alternative way is to implement a List (or Collection) that lazily restores fractions of its data as needed. This can be nicer to use, but is quite a lot more complicated to build (correctly!). Also, if memory constraints are important, then you’ll need to add a mechanism to discard previously-restored objects.