Question

I have an app running on a backend instance. It has 11 tasks. The first one is started by /_ah/start and it, in turn, starts the other ten, the worker tasks. The worker tasks have this structure:

done = False
while not done:
    do_important_stuff()
    time.sleep (30)
    if a_long_time_has_passed():
        done = True

The execution behavior on app engine is the same every time. The scheduling task runs and enqueues the 10 worker tasks. The first seven worker tasks start running, executing correctly. The last three sit in the queue, never running. The task queue app console shows all ten tasks in the queue with seven of them running.

The app also stop responding to HTTP requests, returning 503 status codes with the logs not showing that my http handlers are getting invoked.

The worker task queue is configured with a maximum rate of 1/s and 2 buckets. It’s curious that the admin console shows that the enforced rate is 0.1 sec. Since the tasks run forever, they aren’t returning unsuccessful completion status codes. And the cpu load is negligible. The workers mostly do a URL fetch and then wait 30 seconds to do it again.

The logs are not helpful. I don’t know where to go to find diagnostics that will help me figure it out. I’m testing in a free account. Could there be a limit of 8 tasks executing at one time? I see nothing like that in the documentation, but I’ve run out of ideas. Eventually, I’d like to run even more tasks in parallel.

Thanks for any advice you can give me.

Answer 1

There’s a limit to how many simultaneous requests a backend instance will process, and it sounds like you’re running into that limit.

Alternatives include:

• Use regular task queues rather than ones against a backend
• Start more than one instance of your backend
• Use threading to start threads yourself from the start request, rather than relying on the task queue to do it for you

Note that if your tasks are CPU bound, you’re not going to get any extra benefit from running 10 of them over 5, 2, or maybe even 1.