Question

I have a memory issue with mongoengine (in python).

Let’s say I have a very large amount of custom_documents (several thousands).
I want to process them all, like this:

for item in custom_documents.objects():
    process(item)

The problem is custom_documents.objects() load every objects in memory and my app use several GB …

How can I do to make it more memory wise?
Is there a way to make mongoengine to query the DB lazily (it request objects when we iterates on the queryset)?

Answer 1

According to the docs (and in my experience), collection.objects returns a lazy QuerySet. Your first problem might be that you’re calling the objects attribute, rather than just using it as an iterable. I feel like there must be some other reason your app is using so much memory, perhaps process(object) stores a reference to it somehow? Try the following code and check your app’s memory usage:

queryset = custom_documents.objects
print queryset.count()

Since QuerySets are lazy, you can do things like custom_documents.limit(100).skip(500) as well in order to return objects 500-600 only.