Question

I have a long-running process in a task queue that is causing a DeadlineExceededError because the task can take longer than 10 minutes. As I described in this question, the long-running process has a for loop that sequentially calculates new values in large dictionaries that are used to create kml files. The task currently looks like this:

class longprocess_handler(webapp2.RequestHandler):
def post(self):
#This is currently one task that recursively uses data in dictionaries to 
#produce kml files every few minutes
    for j in range(0, Time):
    # Processes data from dictionaries sequentially in this for loop
    # Sends message to client to request the data.

I would like to make the process into several smaller tasks such as:

class longprocess_handler(webapp2.RequestHandler):
def post(self):
    for j in range(0, Time):
        # Send dictionaries to smaller task 
        CallSmallerTask_handler(dictionaries)
        # Receive dictionaries back from task.  (How do I do this?)
        # Repeat for loop with new dictionaries to call next task.

Is there a way to get data back from a Task so that I can create a loop of smaller tasks that are created sequentially using the result of the previous task? Would I need to store the dictionaries from the previous task in the Datastore and then retrieve them to create the next task? (I am hoping to avoid this because the dictionaries are very large, and I think it may be difficult).

Answer 1

You will have to store the large dictionaries into datastore. Only in the cases where the output of one task is relatively small (not your case), it is possible to pass the output as parameter(s) to the Task’s handler.

One possible optimization is to use ndb or implement your own caching layer on top of the datastore so a good percentage of the read calls will be made from the cache and never reach the datastore. Just keep in mind that storing should be made to the cache and to the datastore since the cache is not 100% reliable.