Home/ Questions/Q 9013703
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-16T03:20:07+00:00

I’m working on a distributed producer/consumer system using a messaging queue. The part that

  • 0

I’m working on a distributed producer/consumer system using a messaging queue. The part that I’m interested in parallelising is the consumer side of it, and I’m happy with what I have for that.

However, I’m not sure what to do about the producer. I only need one producer running at a time since the load of the producing part of my system is not too high, but I want a reliable way of managing it, as in starting, stopping, restarting, and mainly, monitor it so that if the producer host fails another one can pick up.

If it helps, I’m happy with my consumer algorithm, the one that queues jobs, since it’s fault tolerant to be down for a period of time and pick up the stuff that happened during the time it was down.

I’m sure there are tools or at least known patterns to do this and not reinvent the wheel.

I’m using rabbitmq but can use activemq, or even refactor into storm or something like that if needed, my code is not complex so far.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    After a couple of weeks thinking about it, the simplest solution came to mind, and I’m actually very pleased with it, so I’ll share it in case you find it useful, or point out if you think of any downsides, it seems to be working fine so far.

    I have created the simplest table in my DB, called heartbeat, with a single timestamp field called ts, and is meant to have a single row all the time.

    I start all of my potential producers every 5 minutes (quartz), and they do an update of the table if the ts fields is older than now() – 5 minutes. Because the update call is blocking, I’ll have no db threading issues. Now, if the update returns > 0 it means that it actually modified the value of ts, and then I execute the actual producing code (queue jobs). If the update returns 0, it did not modify the table, because someone else did less than 5 minutes ago, and therefore this producer won’t do anything until it checks again in 5 minutes.

    Obviously the 5 minutes value is configurable, and this allows for a very neat upgrade with small changes to be able to execute several producers at the same time, if I ever had that need.

    • 0