Question

I have some scripts, using pika.SelectConnection to communicate with RabbitMq server.

Is there anyway to make those scripts trying automatically reconnect to rmq server, in case when server gone down, while script working?

Answer 1

I have lots of scripts which use the following approach to reconnecting:

 while True:
     try:
         mqExch.channel.connection.drain_events(timeout=25)
     except socket.timeout:
         hb.beat() # let our supervisor know we are not hung

It’s not using pika, but Kombu, nevertheless the principles can be applied. The drain_events method is the core of message consuming, i.e. it loops forever receiving messages and running a callback to process the message. As you can see here I have the low level socket library timing out every 25 seconds. With some libraries I had to patch a couple of lines of code to get this behaviour to work without crashing inside the module.

In any case, the heartbeat that is sent by hb.beat is monitored by a supervisor process and this process will kill the script if there are too many failures in too short a time span. After killing the script, the supervisor restarts it. This has worked well in dealing with intermittent network errors or MQ broker restarts. While I could have made my script do the reconnecting, it was simpler to kill and restart.