Question

I am trying to populate a database by creating entity objects and calling persist() using Hibernate as JPA provider.

At the beginning, the database is empty, then I am creating entity instances (100,000 instances) in a loop, and persist each one.

This operation take like 10 minutes to inject all the entities into the database.
I am using free tire amazon cloud server machine, 600 MB of RAM memory.

The loop is something like the follow:

@Transactional
void populateDataBase()
{
    for(int i=0; i<100000; i++)
    {
        MyEntityPK entityInstancePK = new MyEntityPK(idfield1, idField2, idField3);

        MyEntity entityInstance = new MyEntity();

        entityInstance.setId(entityInstancePK);

        entityInstance.setField1(integerValue);
        entityInstance.setField2(integerValue);

        entityInstance.persist();
    }
}

If I comment the persist() line, the loop finish in few seconds. When I uncomment the entityInstance.persist(); line, the loop take like 10 minutes to finish.

Is this operation is time and memory consuming? or it should not be fast?
Is calling merge() instead of persist() is better in this case? and quicker maybe?

Answer 1

Yeah you need to check out the documentation on batch processing.

You want to do something more like

for ( int i=0; i<100000; i++ ) {
    Customer customer = new Customer(.....);
    session.save(customer);
    if ( i % 20 == 0 ) { //20, same as the JDBC batch size
        //flush a batch of inserts and release memory:
        session.flush();
        session.clear();
    }
}