Question

I want to retrieve the first 500 results from a large database that match a given filter query.

At the moment I’m using the following (terribly inefficient) method..

results = Entries.objects.filter(text__icontains="somequery")[0:500]

But I think this query loads the entire database in memory and then truncates the results. It’s terribly slow.

Is there a more elegant way to do this? Thanks!

Answer 1

That’s the way to do it.

The SQL generated uses LIMIT so it’s not loading the entire database into memory and being python sliced.

Note that you can see what SQL django is writing by using django.db.connection.queries
http://docs.djangoproject.com/en/dev/faq/models/#how-can-i-see-the-raw-sql-queries-django-is-running

But a lesser known trick is to print a queryset.query or call sql = queryset.query.__str__()

>>> results = Entries.objects.filter(text__icontains="somequery")[0:500]
>>> print results.query
SELECT ... FROM ... WHERE ... LIKE ... LIMIT 500