Home/ Questions/Q 3443992
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-18T08:51:35+00:00

I need to iterate over large collection (3 * 10^6 elements) in Django to

  • 0

I need to iterate over large collection (3 * 10^6 elements) in Django to do some kind of analysis that can’t be done using single SQL statement.

  • Is it possible to turn off collection caching in django? (Caching all the data is not to be acceptable data has around 0.5GB)
  • Is it possible to make django fetch collection in chunks? It seems that it tries to pre fetch whole collection in to the memory and then iterate over it. I think that observing the speed of execution:
    • iter(Coll.objects.all()).next() – this takes forever
    • iter(Coll.objects.all()[:10000]).next() – this takes less than a second
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Use QuerySet.iterator() to walk over the results instead of loading them all first.

    • 0