Question

http://twitter.com/#!/ladygaga

When ladygaga tweet 1 message, does it mean to insert 1 data record for EACH of her followers (total 12,221,751)? So totally 12,221,751 records are inserted?

Any clues in designing such a social feed system?

——————————- Edit line ——————————-

Real issue:

Performing SELECT tweet FROM Tweets IN ([FollowingIDs]) is not possible in google app engine, which limiting to a maximum of 30 items in the IN clause

While in app engine it actually means, performing 30 queries in parallel, which is not very wise to do so I guess.

Even if I am allowed to overtake the 30 limits,
what if I am subscribing to 10000 people? I am not sure if there are any performance issues to do it in MYSQL or any other kind of database infrastructure using the “IN clause”
(the bigtable of app engine is different from MYSQL)

So it is better to use the IN clause to query?
or setting up a UserFeed table for storing the feed relationship?
or 3rd method?

Database/SQL guru please help

Answer 1

Please see this talk from Google I/O 2009 to see how to handle these sort of cases on App Engine with a ‘fan out’ data structure.