Question

We want to implement a “News feed” where a user can see messages
broadcasted by her friends, sorted with newest message first. But the
feed should reflect changes in her friends list. (If she adds new
friends, messages from those should be included in the feed, and if
she removes friends their messages should not be included.) If we use
the pubsub-test example and attach a recipient list to each message
this means a lot of manipulation of the message recipients lists when users
connect and disconnect friends.

We first modeled publish-subscribe “fan out” using conventional RDBMS
thinking. It seemed to work at first, but then, since the IN operator
works the way it does, we quickly realized we couldn’t continue on
that path. We found Brett Slatkin’s presentation from last years
Google I/O and we have now watched it a few times but it isn’t clear to
us how to do it with “dynamic” recipient lists.

What we need are some hints on how to “think” when modeling this.

Answer 1

Pasting the answer I got for this question in the Google Group for Google App Engine http://groups.google.com/group/google-appengine/browse_thread/thread/09a05c5f41163b4d# By Ikai L (Google)

A couple of thoughts here:

• is removing of friends a common event? similarly, is adding of
  friends a common event? (All relative,
  relative to “reads” of the news feed)

• From what I remember, the only way to make heavy reads scale is to write
  the data multiple times in peoples’
  streams. Twitter does this, from what
  I remember, using a “eventually
  consistent” model. This is why your
  feed will not update for several
  minutes when they are under heavy
  load. The general consensus, though,
  is that a relational, normalized
  model simply will not work.

• the Jaiku engine is open source for your study:
  http://code.google.com/p/jaikuengine.
  This runs on App Engine Hope these
  help when you’re considering a design.