Question

I thought I understood the concept of cursors and queries based on http://code.google.com/appengine/docs/python/datastore/queryclass.html but apparently it is not so based on what I’m doing in code.

I have 310 items in my datastore and I want to iterate over them in small batch sizes of 100:

query = Event.all()
query.order("__key__")

batch_size = 100;

# expecting this to return the 1st 100 of the 310 items    
results = query.fetch(limit=batch_size)

logging.info("count 1: %d, results: %d" % (query.count(), len(results)))
# reports: count 1: 310, results: 100

for item in results:
    print item # this will print items 1-100, which is expected

# Move to the next batch block
cursor = query.cursor();
query.with_cursor(cursor);            

results = query.fetch(limit=batch_size)
logging.info("count 2: %d, results: %d" % (query.count(), len(results)))
# reports: count 2: 0, results: 0 
# but was expecting to move to item 101

How can I iterate through all of my entities in batches of a 100? Does 'query.cursor()' return the cursor that’s at the end of the 1st batch_size or the start of that block?

Answer 1

The .count() is indeed foiling you. To see why, create a second, identical query, apply the saved cursor to it, and see what happens.

By the way, __key__ order is implicit.