Question

I need to develop a realtime recent activity feed in django (with AJAX long-polling), and I’m wondering what’s the best strategy for the server-side.

Pseudocode:

def recent_activity_post_save():
    notify_view()

[in the view]
while not new_activity():
    sleep(1)
return HttpResponse(new_activity())

The first thing that comes in mind is querying the DB every second. Not feasible. Other options:

1. using the cache as a notification service
2. using a specialized tool, like Celery (I’d rather not do it, because it seems like overkill)

What’s the best way to go here?

Answer 1

I would suggest keeping it simple…

Create a database table to store your events, insert into that table when appropriate, then just implement a simple ajax polling technique to hit the server every x seconds on the client side.

I have concerns with other solutions considering using a push-notification approach or using a noSql data store. It’s a whole lot more complicated than a traditional pull-notification system using the tools that are built in to the Django framework, and except for very rare exceptions, is overkill. Unless you specifically require a strict real-time solution, keep it simple and use the tools that already exist in the framework, and for people with objections based on database or network performance, all I have to say is that premature optimization is the root of all evil.

Build a model that contains recent activity data specific to your application then, whenever your application does something that should log new activity you can just insert into this table.

Your view would simply be like any other view, pulling the top x rows from this RecentActivity table (optionally based on query parameters and whatever).

Then, on the client side, you’d just have a simple ajax poller hitting your view every x seconds. There is no shortage of complicated plugins and technologies you can use, but writing your own isn’t that complicated either:

function simplePoll() {
  $.get("your-url", {query-parameters}, function(data){
    //do stuff with the data, replacing a div or updating json or whatever
    setTimeout(simplePoll, delay);
  });
}

My opinion is that performance issues aren’t really issues until your site is successful enough for them to be an issue. A traditional relational database can scale up fairly well until you start reaching the level of success like Twitter, Google, etc. Most of us aren’t at that level 🙂