Question

How can I NOT have the full result set from mysql copied into memory in django? I’m itterating over a BIG table which is blowing up my ram when the query copies.

It seems that mysql uses SSCursor for this purpose, and I can do all that with the mysql low level interface, but is there a Django way to do it?

Answer 1

This will get 500 items each iteration:

count = Model.objects.count()
chunk = 500
for i in range((count/chunk)+1):
    objs = list(Model.objects.all()[i*chunk:(i+1)*chunk])

You might also want to have a look at the QuerySet method iterator().