Question

I need to implement a messaging system on a web application. It’s powered by PHP and MySQL and will allow registered users to message one another.

So here’s my worry: What will happen after i let the site run for two years and it gets thousands of users with dozens of friends who will each send each other millions of messages over the time?

How are the “big boys” of the IT industry handling those problems, for example Facebook? Surely their messages cannot all fit into one table (MySQL or otherwise)… But even transferring older messages in (one) archive table won’t work forever.

I will in no way build a website that will be even remotely comparable to Facebook, but i still want a solution that won’t bug out after a year or two.

Answer 1

Make a search for Max Value for SQL BIGINT. This script used to overflow when I wrote it a few years ago on a 32-bit machine. It’s much less of a problem now!

http://dev.mysql.com/doc/refman/5.0/en/numeric-type-overview.html

http://www.laprbass.com/RAY_integer_overflow.php

<?php // RAY_integer_overflow.php
error_reporting(E_ALL);
echo "<pre>\n";

// CONNECTION AND SELECTION VARIABLES FOR THE DATABASE
$db_host = "localhost"; // PROBABLY THIS IS OK
$db_name = "??";        // GET THESE FROM YOUR HOSTING COMPANY
$db_user = "??";
$db_word = "??";

// OPEN A CONNECTION TO THE DATA BASE SERVER
// MAN PAGE: http://us2.php.net/manual/en/function.mysql-connect.php
if (!$dbcx = mysql_connect("$db_host", "$db_user", "$db_word"))
{
   $errmsg = mysql_errno() . ' ' . mysql_error();
   echo "<br/>NO DB CONNECTION: ";
   echo "<br/> $errmsg <br/>";
}

// SELECT THE MYSQL DATA BASE
// MAN PAGE: http://us2.php.net/manual/en/function.mysql-select-db.php
if (!$db_sel = mysql_select_db($db_name, $dbcx))
{
   $errmsg = mysql_errno() . ' ' . mysql_error();
   echo "<br/>NO DB SELECTION: ";
   echo "<br/> $errmsg <br/>";
   die('NO DATA BASE');
}

// CREATE A TABLE AND ALTER IT TO A HIGH INDEX NUMBER
$sql = "CREATE TEMPORARY TABLE noise ( id BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY ) ENGINE=MEMORY";
if (!$res = mysql_query($sql)) die(mysql_error());

$sql = "ALTER TABLE noise AUTO_INCREMENT = 2147483646";
if (!$res = mysql_query($sql)) die(mysql_error());


// INSERT DATA TO ADD TO THE AUTO_INCREMENT INDEX
$kount = 0;
while ($kount < 15)
{
   $sql = "INSERT INTO noise () VALUES ()";
   if (!$res = mysql_query($sql)) die(mysql_error());
   $nid = mysql_insert_id($dbcx);
   var_dump($nid);
   $kount++;
}

According to my rough calculations, if you’re aggregating messages at the rate of 1,000 per second and you’re using UNSIGNED BIGINT for your AUTO_INCREMENT key, you will experience an overflow in about 570,000,000 years. So it follows, more or less, that if you’re collecting the messages at the rate of 1,000,000,000 per second, you’ll have a problem in shorter order – perhaps around 570 years from now. But the good news is that if you’re collecting messages at the rate of even 100,000 per second, the VC community will be throwing money at you!