Question

This is probably a truly basic thing that I’m simply having an odd time figuring out in a Python 2.5 app.

I have a process that will take roughly an hour to complete, so I made a backend. To that end, I have a backend.yaml that has something like the following:

-name: mybackend
 options: dynamic
 start: /path/to/script.py

(The script is just raw computation. There’s no notion of an active web session anywhere.)

On toy data, this works just fine.

This used to be public, so I would navigate to the page, the script would start, and time out after about a minute (HTTP + 30s shutdown grace period I assume, ). I figured this was a browser issue. So I repeat the same thing with a cron job. No dice. Switch to a using a push queue and adding a targeted task, since on paper it looks like it would wait for 10 minutes. Same thing.

All 3 time out after that minute, which means I’m not decoupling the request from the backend like I believe I am.

I’m assuming that I need to write a proper Handler for the backend to do work, but I don’t exactly know how to write the Handler/webapp2Route. Do I handle _ah/start/ or make a new endpoint for the backend? How do I handle the subdomain? It still seems like the wrong thing to do (I’m sticking a long-process directly into a request of sorts), but I’m at a loss otherwise.

Answer 1

So the root cause ended up being doing the following in the script itself:

models = MyModel.all()
for model in models:
    # Magic happens

I was basically taking for granted that the query would automatically batch my Query.all() over many entities, but it was dying at the 1000th entry or so. I originally wrote it was computational only because I completely ignored the fact that the reads can fail.

The actual solution for solving the problem we wanted ended up being “Use the map-reduce library”, since we were trying to look at each model for analysis.