Question

Let’s say you have a search form, with multiple select fields, let’s say a user selects from a dropdown an option, but before he submits the data I need to display the count of the rows in the database .

So let’s say the site has at least 300k(300.000) visitors a day, and a user selects options from the form at least 40 times a visit, that would mean 12M ajax requests + 12M count queries on the database, which seems a bit too much .

The question is how can one implement a fast count (using php(Zend Framework) and MySQL) so that the additional 12M queries on the database won’t affect the load of the site .

One solution would be to have a table that stores all combinations of select fields and their respective counts (when a product is added or deleted from the products table the table storing the count would be updated). Although this is not such a good idea when for 8 filters (select options) out of 43 there would be +8M rows inserted that need to be managed.

Any other thoughts on how to achieve this?

p.s. I don’t need code examples but the idea itself that would work in this scenario.

Answer 1

I would suggest a separate table that caches the counts, combined with triggers.

In order for it to be fast you make it a memory table and you update it using triggers on the inserts, deletes and updates.

pseudo code:

CREATE TABLE counts (
  id unsigned integer auto_increment primary key
  option integer indexed using hash key
  user_id integer indexed using hash key
  rowcount unsigned integer
  unique key user_option (user, option)
) engine = memory

DELIMITER $$

CREATE TRIGGER ai_tablex_each AFTER UPDATE ON tablex FOR EACH ROW
BEGIN
  IF (old.option <> new.option) OR (old.user_id <> new.user_id) THEN BEGIN
    UPDATE counts c SET c.rowcount = c.rowcount - 1 
      WHERE c.user_id = old.user_id and c.option = old.option; 
    INSERT INTO counts rowcount, user_id, option  
      VALUES (1, new.user_id, new.option)
      ON DUPLICATE KEY SET c.rowcount = c.rowcount + 1; 
  END; END IF;
END $$

DELIMITER ;

Selection of the counts will be instant, and the updates in the trigger should not take very long either because you’re using a memory table with hash indexes which have O(1) lookup time.

Links:
Memory engine: http://dev.mysql.com/doc/refman/5.5/en/memory-storage-engine.html
Triggers: http://dev.mysql.com/doc/refman/5.5/en/triggers.html