Question

I’m making a micro-blogging website. The users can follow each other. I’ve to make stream of posts (activity stream) for the current user ( $userid ) based on the users the current user is following, like in Twitter. I know two ways of implementing this. Which one is better?

Tables:

Table: posts
Columns: PostID, AuthorID, TimeStamp, Content

Table: follow
Columns: poster, follower

The first way, by joining these two tables:

select `posts`.* from `posts`,`follow` where `follow`.`follower`='$userid' and 
`posts`.`AuthorID`=`follow`.`poster` order by `posts`.`postid` desc

The second way is by making an array of users the $userid is following (posters), then doing php implode on this array, and then doing where in:

One thing I’ll like to tell here that I’m storing the the number of users a user is following in the `following` record of the `user` table, so here I’ll use this number as a limit when extracting the list of posters – the ‘followingList’:

function followingList($userid){
    $listArray=array();
    $limit="select `following` from `users` where `userid`='$userid' limit 1";
    $limit=mysql_query($limit);
    $limit=mysql_fetch_row($limit);
    $limit= (int) $limit[0];
    $sql="select `poster` from `follow` where `follower`='$userid' limit $limit";
    $result=mysql_query($sql);
    while($data = mysql_fetch_row($result)){
        $listArray[] = $data[0];
    }
    $posters=implode("','",$listArray);
    return $posters;
}

Now I’ve a comma separated list of user IDs the current $userid is following.

And now selecting the posts to make the activity stream:

$posters=followingList($userid);
$sql = "select * from `posts` where (`AuthorID` in ('$posters')) 
order by `postid` desc";

---

Which of the two methods is better?
And can knowing the total number of following (number of users the current user is following), make things faster in the first method as it’s doing in the second method?
Any other better method?

Answer 1

The first important point is that PHP is good at building pages but very bad are managing data, everything manipulated by PHP will fill the memory and no special behavior can be applied in PHP to prevent using to much memory, except crashing.

On the other side the datatase job is to analyse relation between the tables, real number used by the query (cardinality of indexes and statictics on rows and index usage in fact), and a lot of different mechanism can be choosen by the engine depending on the size of data (merge joins, temporary tables, etc). That means you could have 256.278.242 posts and 145.268 users, with 5.684 average followers the datatabase job would be to find the fastest way to give you an answer. Well, when you hit really big numbers you’ll see that all databases are not equal, but that’s another problem.

On the PHP side Retrieving the list of users from the fisrt query coudl became very long (with a big number of followed users, let’s say 15.000. Simply building the query string with 15 000 identifiers inside would take a quite big amount a memory. Trasnferring this new query to the SQL server would also be slow. It’s definitively the wrong way.

Now be careful of the way you build your SQL request. A request is something you should be able to read from the top to the end, explaining what you really want. This will help the SQL (good) engine in choosing the right solution.

select `posts`.* 
from `posts`
  INNER JOIN `follow` ON posts`.`AuthorID`=`follow`.`poster`
where `follow`.`follower`='@userid' 
order by `posts`.`postid` desc
LIMIT 15

Several remarks:

• I have used an INNER JOIN.I want an INNER JOIN, let’s write it, it will be easier to read for me later and it should be the same for the query analyser.
• if @userid is an int do not use quotes. Please use ints for identifiers (this is really faster than strings). And on the PHP side cast the int "SELECT ..." . (int) $user_id ." ORDER ... or use query with parameters (This is for security).
• I have used a LIMIT 15, maybe an offset could be used as well, if you want to show some pagination control around the posts. Let’s say this query will retrieve 15.263 documents from my 5.642 folowwed users, you do not want, and the user do not want, to show theses 15.263 documents on a web page. And knowing with $limit that the number is 15.263 is a good thing but certainly not for a request limit. You know this number, but the database may know it as well if it has a good query analyser and some good internal statistics.

The request limit has several goals
1. Limit the size of data transfered from the database to your PHP script
2. Limit the memory usage of your PHP script (an array with 15.263 documents containg some HTMl stuff… ouch)
3. Limit the size of the final user output (and get a faster response)