I already read the respective article in Celery docs (Cookbook :: Creating Tasks), however I didn’t completely like the solution suggested. I have very long running tasks (from 15 minutes to 1 hour or possibly even more) and thus choosing some LOCK_EXPIRE setting doesn’t seem reliable: if it turns out too small, I get two tasks running simultaneously and if it is too big and a celery process dies for some reason, I’ll need to manually delete the locking cache key.
The idea that comes to my mind is that I could use two different locks: one created with fcntl.lockf (for different celery worker processes) and another one with threading.Lock() (when the task gets executed in the same celery process). However having two locks seems somewhat excessive.
So, my question is: what would be the simplest solution for my problem? It doesn’t have to be cross-platform and can support only Linux.
Just use one lock without expiration, and create a periodic “investigator” Task that checks whether everything is ok, that is, if another Task died unexpectedly, it fixes the lock issue. As a plus, take a look here.