Question

What packages should I look at for writing a python daemon and processing jobs? Also, what do I need to do for a python daemon?

Answer 1

Your question is a bit ambiguous, but I’m assuming you mean you would like to write a python daemon that will process jobs that get thrown in a queue. If not, please say as much. 🙂

I’ve heard a lot of great things about redis. The folks at github built resque as a job processing daemon for Ruby. If you’re language flexible, you could just use that, but if you’re not, you could emulate it in as much or as little depth as you like making use of redis as your queue system. Depending on how pluggable and extensible you need it to be, this could be a really simple thing to implement.

Another option I ran across after some more googling is redqueue. It looks like it might already implement most of a job queue.

If you’re using django, you may wish to consider the Celery project. It’s a job queue system based on RabbitMQ which is yet another queuing server with excellent reviews.

As far as creating a daemon in python, there are a number of options. You can look at this page on activestate, which is a good start. Better yet, you can use python-daemon to do it all for you. But if you use one of the above options or beanstalkd as recommended by mczepiel, you probably won’t have to make your process run as a daemon.