Question

I have an educational site that serves up curriculum based upon Grade level.

1. The GradeLevel table stores all of the possible Grade Levels. Then I have LessonCategories and LessonCurriculum tables.

2. The GradeLevel table has reverse relationships established with the curriculum and categories tables.

3. I loop through each grade in the GradeLevel table, (8 grades), and grab the respective, curriculum and categories along the way.

4. Upon completion, I stuff all of the collected curriculum and categories into a list and and pass that to my template.

Now, the problem is that Django is evaluating each query AT LEAST twice. Once for the initial request and the second when I put it into the list. (I am using itertools to chain the results. Itertools is causing the queries to be run again.) This is having the adverse effect of slowing my server down to a crawl.

My question is if someone could take a look at my models and queries and make a suggestion as to a better way to query as to avoid and/or mitigate this MAJOR performance bottle-neck.

GradeLevel Model:

class GradeLevel(models.Model):
    title = models.CharField('Grade',max_length=10, null=True, blank=True, db_index=True)
    fullname = models.CharField('Description',max_length=100, null=True, blank=True, db_index=True)

LessonCategory Model:

class LessonCategory(models.Model):
    title = models.CharField(max_length=255, null=True, blank=True, db_index=True)
    ...
    gradelevel = models.ManyToManyField(GradeLevel, related_name='grade_cats', null=True, blank=True)

LessonCurriculum:

class LessonCurriculum(models.Model):         
    title = models.CharField(max_length=255, null=True, blank=True, db_index=True)
    ...
    gradelevel = models.ManyToManyField(GradeLevel, related_name='grade_curriculum', null=True, blank=True)

My View:

from itertools import chain
from operator import attrgetter

def my_view(request):
    grade_pk =  GradeLevel.objects.prefetch_related().get(title='pre-k')
    grade_pk_categories = grade_pk.grade_cats.filter(active=True,featured=True)
    grade_pk_galleries = grade_pk.grade_curriculum.filter(active=True,featured=True)

    grade_k =  GradeLevel.objects.prefetch_related().get(title='k')
    grade_k_categories = grade_k.grade_cats.filter(active=True,featured=True)
    grade_k_galleries = grade_k.grade_curriculum.filter(active=True,featured=True)

    grade_1 =  GradeLevel.objects.prefetch_related().get(title='1')
    grade_1_categories = grade_1.grade_cats.filter(active=True,featured=True)
    grade_1_galleries = grade_1.grade_curriculum.filter(active=True,featured=True)

    grade_2 =  GradeLevel.objects.prefetch_related().get(title='2')
    grade_2_categories = grade_2.grade_cats.filter(active=True,featured=True)
    grade_2_galleries = grade_2.grade_curriculum.filter(active=True,featured=True)

    grade_3 =  GradeLevel.objects.prefetch_related().get(title='3')
    grade_3_categories = grade_3.grade_cats.filter(active=True,featured=True)
    grade_3_galleries = grade_3.grade_curriculum.filter(active=True,featured=True)

    grade_4 =  GradeLevel.objects.prefetch_related().get(title='4')
    grade_4_categories = grade_4.grade_cats.filter(active=True,featured=True)
    grade_4_galleries = grade_4.grade_curriculum.filter(active=True,featured=True)

    grade_5 =  GradeLevel.objects.prefetch_related().get(title='5')
    grade_5_categories = grade_5.grade_cats.filter(active=True,featured=True)
    grade_5_galleries = grade_5.grade_curriculum.filter(active=True,featured=True)

    grade_6 =  GradeLevel.objects.prefetch_related().get(title='6')
    grade_6_categories = grade_6.grade_cats.filter(active=True,featured=True)
    grade_6_galleries = grade_6.grade_curriculum.filter(active=True,featured=True)

    grade_7 =  GradeLevel.objects.prefetch_related().get(title='7')
    grade_7_categories = grade_7.grade_cats.filter(active=True,featured=True)
    grade_7_galleries = grade_7.grade_curriculum.filter(active=True,featured=True)

    grade_8 =  GradeLevel.objects.prefetch_related().get(title='8')
    grade_8_categories = grade_8.grade_curriculum.filter(active=True,featured=True)
    grade_8_galleries = grade_8.grade_curriculum.filter(active=True,featured=True)

    gallery_list = list(set(sorted(chain(grade_pk_categories,grade_pk_galleries,grade_k_categories,grade_k_galleries,grade_1_categories,grade_1_galleries,grade_2_categories,grade_2_galleries,grade_3_categories,grade_3_galleries,grade_4_categories,grade_4_galleries,grade_5_categories,grade_5_galleries,grade_6_categories,grade_6_galleries,grade_7_categories,grade_7_galleries,grade_8_categories,grade_8_galleries), key=attrgetter('display_order'))))

Answer 1

What you want to use is chain.from_iterable. This will prevent the queries from being run more than once.

gallery_list = tuple(chain.from_iterable(grade_pk_categories,grade_pk_galleries,grade_k_categories,grade_k_galleries,grade_1_categories,grade_1_galleries,grade_2_categories,grade_2_galleries,grade_3_categories,grade_3_galleries,grade_4_categories,grade_4_galleries,grade_5_categories,grade_5_galleries,grade_6_categories,grade_6_galleries,grade_7_categories,grade_7_galleries,grade_8_categories,grade_8_galleries))

You also want to be sorting after you run your items through a set, not before.

gallery_list = list(sorted(frozenset(gallery_list), key=attrgetter('display_order')))

I would reconsider how you are going about fetching all of this data. You are doing two query for each grade level when you could probably replace it with one or two queries for all of the data.

At the very least you could use a for loop to reduce the amount of code you have:

grade_titles = ['pre-k', 'k', '1', '2', '3', '4', '5', '6', '7', '8']
gallery_list = []
for grade_title in grade_titles:
    grade = GradeLevel.objects.prefetch_related().get(title=grade_title)
    grade_categories = grade_pk.grade_cats.filter(active=True,featured=True)
    grade_galleries = grade_pk.grade_curriculum.filter(active=True,featured=True)
    gallery_list.extend(grade_categories)
    gallery_list.extend(grade_galleries)
gallery_list = list(sorted(frozenset(gallery_list), key=attrgetter('display_order')))