Question

I have n parallel tasks running. When all these tasks are done, I want to create another task. What is the best way to do this ? Since a datastore operation is slower, I used memcache. I keep incrementing the counter by 1 when each task is finished. I keep checking the counter for n. Since memcache is not reliable, the counter value is lost sometimes. Any other solutions?

Answer 1

Since you can’t afford to lose your counter, you need to use the datastore to ensure that your counter isn’t lost (e.g., due to an unexpected memcache eviction). However, you could also store the value in memcache – that way you can still get fast reads of the counter from memcache (and go to the datastore to get the count if the memcache entry has expired).

If you have so many tasks running in parallel that incrementing the datastore counter is too slow, then consider using a sharded counter to relieve this bottleneck.