Question

I’ve got 3 parts to this question:

I have an application where users create objects that other users can update within 5 minutes. After 5 minutes, the objects time out and are invalid. I’m storing the objects as entities. To do the timeout, I have a cron job that runs once a minute to clear out the expired objects.

Most of the time right now, I don’t have any active objects. In this case, the mapreduce handler checks the entity it gets, and does nothing if it’s not active, no writes. However, my free datastore write quota is running out from the mapreduce calls after about 7 hours. According to my rough estimate, it looks like just running mapreduce causes ~ 120 writes/call. (Rough math, 60 calls/hr * 7 hr = 420 calls, 50k ops limit / 420 calls ~ 120 writes/call)

Q1: Can anyone verify that just running mapreduce triggers ~120 datastore writes?

To get around it, I’m checking the datastore before I kick off the mapreduce:

def cronhandler():
    count = model.all(keys_only=True).count(limit=1000) 
    if count:
        shards = (count / 100) + 1;
        from mapreduce import control
        control.start_map("Timeout open objects",
                      "expire.maphandler",
                      "expire.OpenOrderInputReader",
                      {'entity_kind' : 'model'},
                      shard_count=shards)
    return HttpResponse()

Q2: Is this the best way to avoid the mapreduce-induced datastore writes? Is there a better way to configure mapreduce to avoid extraneous writes? I was thinking potentially it was possible with a better custom InputReader

Q3: I’m guessing more shards result in more extraneous datastore writes from mapreduce bookkeeping. Is limiting shards by the expected number of objects I need to write appropriately?

Answer 1

I’m assuming what I’ve done is the best way to go about doing things. It looks like the Mapreduce API uses the datastore to keep track of the jobs launched and synchronize workers. By default the API uses 8 workers. Reducing the number of workers reduces the number of datastore writes, but that reduces wall time performance as well.