Question

I’m working with a code base that is new to me, and it uses iBatis.
I need to update or add to an existing table, and it may involve 20,000+ records.
The process will run once per day, and run in the middle of the night.

I’m getting the data from a web services call. I plan to get the data, then populate one model type object per record, and pass each model type object to some method that will read the data in the object, and update/insert the data into the table.

Example:

ArrayList records= new ArrayList();
Foo foo= new Foo();
foo.setFirstName("Homer");
foo.setLastName("Simpson");
records.add(foo);
//make more Foo objects, and put in ArrayList.

updateOrInsert(records); //this method then iterates over the list and calls some method that does the updating/inserting

My main question is how to handle all of the updating/inserting as a transaction. If the system goes down before all of the records are read as used to update/insert the table, I need to know, so I may go back to the web services call and try again when the system is ok.

I am using Java 1.4, and the db is Oracle.

Answer 1

I would highly recommend you consider using spring batch – http://static.springsource.org/spring-batch/

The framework provides lot of the essential features required for batch processing – error reporting, transaction management, multi-threading, scaling, input validation.

The framework is very well designed and very easy to use.

The approach you have listed might not perform very well, since you are waiting to read all objects, storing them all in memory and then inserting in the database.

You might want to consider designing the process as follows:

1. Create a cache capable of storing 200 objects
2. Invoke the webservice to fetch the data
3. Create an instance of an object, validate and store the data in the object’s fields
4. Add the object to the cache.
5. When the cache is full, perform a batch commit of the objects in the cache to the database
6. Continue with step 1

SpringBatch will allow you to perform batch commits, control the size of the batch commits, perform error handling when reading input – in your case retry the request, perform error handling while writing the data to the database.

Have a look at it.