Home/ Questions/Q 4267370
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-21T06:53:25+00:00

I am working on a product, that requires crawling APIs and computing and returning

  • 0

I am working on a product, that requires crawling APIs and computing and returning results.
I am currently using Django as product. I had a detailed look at Documentation of Celery and RabbitMQ and queues in general.

What I want to achieve is:
1. Monitoring / administration of each API called for different vendor.
2. Multiple queues with Multiple consumers.
3. Real time Re-Scheduling in Queues.

The main problem here is that in our entire cycle the main time consuming process is API requests taking anywhere between 1 sec to 20 sec. Is it a good idea to use celery / RabbitMQ. I cannot use a Rabbit queue with acknowledgement to make a API request and wait for it to get over while keeping others waiting in queues, queues without acknowledgement would not help as keeping track of failed requests and retrying them would not be possible.
So, how would I use queues to work around this issue?

Second For monitoring I had a look at this http://www.rabbitmq.com/management.html, the management n monitoring plugin, but I cannot find a way to get the information about processed messages in the queue which is very important in my case.
Is there any way I can get this info for RabbitMQ.

Should I use any other queue like ActiveMQ or ZeroMQ to get better monitoring plugins specially to get info about processed messages from the queue.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    I’ve used Beanstalk for a while now with great success.

    Server: https://github.com/kr/beanstalkd
    Client Library: https://github.com/PeterScott/beanstalkc
    Django Helper: https://github.com/jonasvp/django-beanstalkd
    Django Admin App: https://github.com/humanfromearth/django-beanstalk

    I think Beanstalk has many of the options you are looking for. For example:
    1. You can set TTR (time to run) for how long a job has to run before it is released back into the queue
    2. If a job is nearing the end of its TTR, you can ‘touch’ it to reset the TTR countdown
    3. Beanstalk keeps track of stats at the server, tube, and job level. So you can see how many jobs are active, delayed, total, etc.

    RabbitMQ is very robust, but it’s a heavy weight. I’ve found Beanstalk to be much lighter, flexible, and easier to use.

    • 0