Home/ Questions/Q 251701
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-11T21:37:00+00:00

I have a situation here where I need to distribute work over to multiple

  • 0

I have a situation here where I need to distribute work over to multiple JAVA processes running in different JVMs, probably different machines.

Lets say I have a table with records 1 to 1000. I am looking for work to be collected and distributed is sets of 10. Lets say records 1-10 to workerOne. Then records 11-20 to workerThree. And so on and so forth. Needless to say workerOne never does the work of workerTwo unless and until workerTwo couldnt do it.

This example was purely based on database but could be extended to any system, I believe be it File processing, email processing and so forth.

I have a small feeling that the immediate response would be to go for a Master/Worker approach. However here we are talking about different JVMs. Even if one JVM were to come down the other JVM should just keep doing its work.

Now the million dollar question would be: Are there any good frameworks(production ready) that would give me facility to do this. Even if there are concrete implementations of specific needs like Database records, File processing, Email processing and their likes.

I have seen the Java Parallel Execution Framework, but am not sure if it can be used for different JVMs and if one were to come down would the other keep going.I believe Workers could be on multiple JVMs, but what about the Master?

More Info 1: Hadoop would be a problem because of the JDK 1.6 requirement. Thats bit too much.

Thanks,
Franklin

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    You could also use message queues. Have one process that generates the list of work and packages it in nice little chunks. It then plops those chunks on a queue. Each one of the workers just keeps waiting on the queue for something to show up. When it does, the worker pulls a chunk off the queue and processes it. If one process goes down, some other process will pick up the slack. Simple and people have been doing it that way for a long time so there’s a lot information about it on the net.

    • 0