Question

I have a lot of data in a database(PostgreSQL) and need to process all. My program have threads to process all these data and follows these logic.

1. Get a part of data from database
2. Process
3. Save data

I have doubt about how is best way to do this. I have three ideas:

• Create a manager class that runs in a loop getting data from database and holding a queue of objects to process. Create a process class that runs in a loop getting the object to process from the manager class.

• To de same above, but without the manager class, so the process class will have the queue of objects shared between it and they will be responsible for getting the data from database too.

• A manager class that runs in a loop getting data from database, but it create the process classes with the data to process, so the process class won’t request nothing from the manager. It’s created, processed and destroyed, and not run in a loop.

I don’t know what is better, and if there is another solution more efficient.

Answer 1

You are describing so called manager-worker model. I think that your first description is better.

It pushes data into a queue and multiple workers process it. You can use thread pool for workers. The workers are waiting on queue. Once work is pushed to queue one of the workers takes it immediately. When they are done they can push the result into outgoing queue and yet another thread will send the data to DB. Alternatively each worker can save results himself. It is up to you and depends on your task.

User Excecutors and BlockingQueue for implementation. All you need is in java.util.concurrent package and you can find a lot of tutorials and example in web how to use them.

Good luck.