Question

I am looking for a JPA-solution (vendor-independent) to execute a query in batches. The challenge is to make this performant as well as thread-safe.

Example query:

Query query = em.createQuery("select e from Entity e where e.property in :list");

The list is a collection of size between 1 and 385000. Hence, the requirement to batch this query.

Initial naive approach was to get a sublist from the original list and loop through until done. This was safe and working well except that it was not performant.

Second approach was to load everything from the list onto a temp table (permanent in existence, but used as a temporary table) and then use the original query and join with the temp table. This is definitely performant, but is not thread-safe as I need to clear the temp table after each batch and without having any thread id or something of that sort in the temp table its pretty unsafe (which is at the moment).

I would really appreciate suggestions to arrive at a performant and safe way to tackle this issue.

Thanks

Answer 1

First of all, the query is not valid JPQL, because it doesn’t have a select clause.

Second, it should be where e.property in (:list).

Your strategy of populating a temp table looks fine to me. You could just make it contain an additional uuid column, and generate a new UUID each time you want to perform such a query:

• generate a UUID
• insert all the elements of the list in the table, with the uuid column set to the generated UUID
• execute a query such as select e from Entity e, TempEntity temp where e.property = temp.property and temp.uuid = :uuid
• execute a query to delete all the rows from the temp table (not absolutely necessary): delete from TempEntity temp where temp.uuid :uuid