Question

I have a Gunicorn server running a Django application which has a tendency to crash quite frequently. Unfortunately when it crashes all the Gunicorn workers go down simultaneously and silently bypass Django’s and django-sentry’s logging. All the workers return “Internal Server Error” but the arbiter does not crash so supervisord does not register it as a crash and thus does not restart the process.

My question is, is there a way to hook onto a Gunicorn worker crash and possibly send an email or do a logging statement? Secondly is there a way to get supervisord to restart Gunicorn server that is returning nothing but 500’s?

Thanks in advance.

Answer 1

I highly recommend using zc.buildout. Here is an example using plugin Superlance for supervisord with buildout:

[supervisor]
recipe = collective.recipe.supervisor
plugins =
   superlance
...
programs =
     10 zeo ${zeo:location}/bin/runzeo ${zeo:location}
     20 instance1 ${instance1:location}/bin/runzope ${instance1:location} true
...
eventlisteners =
     Memmon TICK_60 ${buildout:bin-directory}/memmon [-p instance1=200MB]
     HttpOk TICK_60 ${buildout:bin-directory}/httpok [-p instance1 -t 20 http://localhost:8080/]

Which will do http request every 20 seconds and restart process if it fails.

http://pypi.python.org/pypi/collective.recipe.supervisor/0.16