Question

So I’m currently working on adding a recommendation engine to a Django project and need to do some heavy processing (in an external module) for one of my view functions. This significantly slows down page load time because I have to load in some data, transform it, perform my calculations based on parameters sent by the request, and then return the suggestions to the view. This has to be done every time the view is loaded.

I was wondering if there was some way I could have the recommender module load and transform the data in memory, and then wait for parameters to be sent from the view, have calculations run on those parameters and then send it back to the view.

Any help would be greatly appreciated.

Answer 1

Celery is a task queue that reeally excels at this sort of thing.

It would allow you to do something like:

1. user makes request to view
2. view starts an async task that does the heavy lifting, then returns to the user immediately
3. you can poll from javascript to see if your task is done and load the results when it is

Might not quite be the flow you’re looking for but celery is definitetly worth checking out
Celery has a great django package too, extremely easy to use

Rereading your question, i think it would also be possible to create a local webservice around your recommendation engine. On startup it can load all the data into memory, then you can just make requests to it from your django app?