Question

In my Django web app, I have a worker program that is a client of a rate-limited API and is responsible for handling all requests to that API from my server threads. I use my database to store the task queue. Tasks can come in large bunches, or not at all. I’m using an event loop to poll the queue and manage delay in between tasks, in case the rate-limit is exceeded (the limit is dynamic). This all works fine, but the only other thing I want to do is to have a way for the worker to stop hitting the database if the queue goes dry, and a way for my Django app to signal the worker that the queue is no longer dry again.

Schematically, in pseudo-Python it looks like this:

state = NORMAL
delay_time = NORMAL_DELAY

while True:
    sleep(delay_time)

    if state == DORMANT:
        continue

    task = get_next_task() # hits database
    if task is None:
        state = DORMANT
        delay_time = NORMAL_TIME

    try:
        execute(task)
    except RateExceeded:
        delay_time = backoff(delay_time)
    else:
        delay_time = NORMAL_DELAY

# Triggered by web layer
def asynchronous_event():
    state = NORMAL

And I either want an asynchronous event triggered from the web layer that can set state back to NORMAL (which would execute during sleep) or some other lightweight check that won’t add needless looped DB queries.

In a single-machine setup, I could just use signals, but obviously this doesn’t work in a multi-machine setup. I’m trying not to have to run a separate message queue server just for the purpose of this one signal. I’m hosted on Dotcloud, in case that plays into what network-based solutions would work. Ideally, something more or less equivalent in ease of implementation as a signal handler. I’ve looked into ZeroRPC, but I’m not sure how to incorporate it into my event loop.

Any ideas?

Edit

I’m looking into ZeroMQ to solve this problem, but I could use some help. The tricky part is that there will be multiple concurrent webserver instances, and upon redeployment, I need for there to be a smooth transition from one worker to its successor. So, bear with me, because my terminology is probably not correct, it seems to me that the best thing to do is have each worker bind asynchronously to an address as a mailbox, which gets checked in the main loop to wake up from dormant mode. Each worker creates a record in the database of its IP, timestamped with creation date. When submitting a request, the web server publishes a message to all workers. When a worker receives a message, it checks if its the worker with the latest creation date: if so, it processes the message, and if not, it terminates itself.

It seems like a lot of trouble, but I want to get this right because I will likely be using this paradigm elsewhere in my application.

Answer 1

As it turned out, I decided just to hit the database every loop through. But I also decided that ZeroMQ is the way to go if I ever want to put in the effort to make this really efficient. Here’s how it works:

Each worker binds a ZeroMQ subscriber socket and registers itself in a database of workers, which contains the IP address and port of the socket. The web threads publish a DO_TASK message to the most recently registered worker and QUIT messages to any others that might be working.

I’m deploying on Dotcloud, and their support says that using the custom service environment variables and build options will let me open the necessary ports and get the IPs of the worker task instances.