Question

I am writing a deferred task which is intended to construct a file in the blobstore for download. I am modelling the code on the example given in the docs:

http://code.google.com/appengine/articles/deferred.html

The idea is to structure the code so that if there is a DeadlineExceededError the handler can tidy up and kick off a new deferred task to continue later.

What I’d like to know is when exactly can this exception be thrown? Are there any operations which are guaranteed to be atomic and therefore will not be interrupted?

In the example (referenced above) they update a variable called start_key as they finish processing each record, but say the main loop was interrupted between the extending of the to_put and to_delete lists then the data would be wrong, as it would do miss a set of deletes.

If an exception can be raised at any point then it could be halfway through the batch_write, or between the db.put and clearing of the to_put list.

This is logically equivalent to a thread safety problem, to solve it one normally has guaranteed atomic operations and non-atomic operations.

How does this work?

Thanks

Answer 1

A DeadlineExceededError can be thrown literally any time at all. If there were a time when it couldn’t be thrown, an abusive app could simply execute that code in a loop.

You can avoid this several ways:

1. Proactively check how long you’ve been executing for and stop at a good time before you hit the deadline.
2. Put the exception handler somewhere that it can store the state as of the last set of completed operations (eg, discarding anything since the last iteration of the outer loop in which the exception was thrown)
3. Use backends, which do not have deadlines.