Question

I am developing an application that needs to perform lots of batch processing (for example whois request). To ensure best performance, I would like to split the job between different computers. For this, I am planning to write a program which will query the job queue on the main server, fetch one job, processes it and updates the main server with the result. The actual processing will be done by PHP. I program only need to poll the job queue and invoke the local php script. The job needs to run every few seconds, hence I cannot use cron.

Can anyone suggest a programming language that can create such a daemon easily? Is there any program available that already does this?

Thanks

Answer 1

A few notes

1. You can use cron to run every few seconds(using hacks though)
2. You will need some sort of distributed queue to hold your jobs (RabbitMQ is a good one or you can use ZooKeeper)
3. Depending on the queue you pick, there are API’s in many programming languages to remove jobs from the queue.

There are many open source tools that will do similar things, but it will greatly depend on how sophisticated your needs are.

1. Hadoop is a complex product that will let you easily implement this
2. workerpool is a python library that is simple to use, but this is multithreaded and will run a single machine. So it is on the simple end of the spectrum.