Question

I’m pretty new to java and really enjoying learning about it. I have made a program that is working well but takes a bit long when I add more data for it to process. I made it threaded and it really speeded it up alot, but now I’m thinking of trying to speed it up more(obviously more data it has to process the longer it takes). Just an fyi, my program does not share any data between threads,it gets one item of a list and does some math and uploads the result to a database. Ideally, a few work computers get a few items of the list and then do its work and then get more work until its done

I did a bit of research and found queues, and am not sure if its what I need or if there’s something else out there(also I was thinking maintaining integrity/monitor of workers might be too much for me to write as a newbie). I have 4 computers at home(some Linux, mac and windows..but I can install linux vm’s on all nonlinux systems if these solutions are os specific) and wanted to try to get them to start working on this task as well. I thought about creating Java queues that the other clients take a piece and process but I also saw libraries(rabbitmq). I also did briefly look over grid computing.

Is this the way to go or is there a better way? I don’t need any code or anything just want to know what are the solutions to distributing threads or what factors to use when evaluating them.

Answer 1

Just to wrap up – you have already scaled-up, now you want to scale-out. From the top of my head:

• terracotta: you can create a Java Queue that will be automatically distributed across the whole cluster. Basically you run the same application with few threads reading data from the queue. Terracotta magically distributes that queue so it feels like local object.

• hazelcast – similar approach to Terracotta, distributed data structures and executors

• jms – send a message with a piece of work to a JMS queue (queue, again) and have several listeners. Each listener is more-or-less a single thread. Listeners can be set up on different machines.

• hadoop – Map/Reduce Java framework to easily scale out large amounts of data. Typically used to process data and aggregate results.