If I have a loop that enqueues say 100 tasks and each one of those tasks potentially enqueues a task, how can I tell if all tasks have completed?
I’ve thought about this problem using ShardedCounters. Once each task is completed, I could increment a counter then check to see if count == 100. Of course that falls apart with tasks spawning their own tasks unless I get into this recursive counting scenario. I’m not sure it’s a good idea to go down that rabbit hole because it appears the sharded counters are not atomic.
You should take a look at the new AppEngine pipeline API. It’s a way of chaining together expensive tasks, and notifying when they’re complete. There’s a also great video from the Google IO conference which demonstrates it.