Question

I have a function that makes many HTTP requests to collect data and stores it in the datastore. All calls must succeed or else everything must be rolled back. The length of time to execute approaches 10 minutes. The max transaction deadline on app engine appears to be the default 60 seconds.

Is there a way to increase this and thus make the function transactional?

Answer 1

It looks like it’s not possible. From the docs:

Maximum time, in seconds, to wait for Datastore to return a result before aborting with an error. Accepts either an integer or a floating-point value. Cannot be set higher than the default value (60 seconds), but can be adjusted downward to ensure that a particular operation fails quickly (for instance, to return a faster response to the user, retry the operation, try a different operation, or add the operation to a task queue).

https://developers.google.com/appengine/docs/python/datastore/functions#create_transaction_options

However you can trigger a task when and only when a transaction completes successfully, if that’s of any use.

https://developers.google.com/appengine/docs/python/datastore/transactions#Transactional_Task_Enqueuing

You can enqueue a task as part of a Datastore transaction, such that the task is only enqueued—and guaranteed to be enqueued—if the transaction is committed successfully. If the transaction does not get committed, the task is guaranteed not to be enqueued. If the transaction does get committed, the task is guaranteed to be enqueued.