Question

I am trying to output the total content views from my stats table and group by the year… My stats table is INNODB and has 8M lines and growing…

The table is essentially ID, DATE, MAKE, IP, REFERRER (indexes on id,date,make)

Each entry has an auto-incremented ID, the entry date YYYY-MM-DD HH:MM:SS, and a product make like ‘sony’, ‘panasonic’ etc…

I am trying to make a query that does not kill my server that sums up the total content views per year and shows them in order from most viewed to least viewed…(for this year 2011) so that I can use that data to populate a JS chart comparing this year with the past years. I can do this with multiple queries and walking through arrays in PHP but I think there should be a way to get this in one query, but hell if I can figure it out.

Any ideas? Also, am I better to make three independent queries and deal with the results in PHP or can I get this into one query that is more MYSQL efficient.

The output I would like to see (although I cannot seem to make it do this), is simply

MAKE           2009Total   2010Total 2011Total
----           ---------   --------- ---------
Panasonic      800         2345      3456
Sony           998         5346      2956
JVC            1300        1234      1944           

Assume my table has data in it from 2009 to now, I need my array to contain one line per make…

Any help would be appreciated… I am amazed at how fast results like this come back from analytics tools and mine take about 75seconds on 4x Quad-core XEON RAID mysql server… this stats table is not being written to but once a day to dump in the previous day’s stats so I am not sure why my 3 sep queries are so slow… hence my question… maybe a single query won’t be any faster?

Anyway, any help would be appreciated and opinions about speeding up stats queries from a generic view stats table would be welcomed!

Answer 1

I have made an observation. Your query is requesting by year. You should do two things:

1. store the year
2. create a better index (product,year)

Here is how yuou can do so:

CREATE TABLE stats_entry_new LIKE stats_entry;
ALTER TABLE stats_entry_new ADD COLUMN entryyear SMALLINT NOT NULL AFTER date;
ALTER TABLE stats_entry_new ADD INDEX product_year_ndx (product,year);
ALTER TABLE stats_entry_new DISABLE KEYS;
INSERT INTO stats_entry_new
SELECT ID, DATE,YEAR(date),product,IP,REFERRER FROM state_entry;
ALTER TABLE stats_entry_new ENABLE KEYS;
ALTER TABLE stats_entry RENAME stats_entry_old;
ALTER TABLE stats_entry_new RENAME stats_entry;

Now the query looks like this:

SELECT A.product,B.cnt "2009Total",C.cnt "2010Total",D.cnt "2011Total"
FROM
(SELECT DISTINCT product FROM stats_entry) A
INNER JOIN 
(SELECT product,COUNT(1) cnt FROM stats_entry WHERE entryyear=2009 GROUP BY product) B
USING (product)
(SELECT product,COUNT(1) cnt FROM stats_entry WHERE entryyear=2010 GROUP BY product) C
USING (product)
(SELECT product,COUNT(1) cnt FROM stats_entry WHERE entryyear=2011 GROUP BY product) D
USING (product);

Now to be fair, if you do not want to add a year to the table then you still have to make an index

ALTER TABLE stats_entry ADD INDEX product_date_ndx (product,date);

Your query looks like this now

SELECT A.product,B.cnt "2009Total",C.cnt "2010Total",D.cnt "2011Total"
FROM
(SELECT DISTINCT product FROM stats_entry) A
INNER JOIN 
(SELECT product,COUNT(1) cnt FROM stats_entry
WHERE date >= '2009-01-01 00:00:00'
AND date <= '2009-12-31 23:59:59'
GROUP BY product) B
USING (product)
(SELECT product,COUNT(1) cnt FROM stats_entry
WHERE date >= '2010-01-01 00:00:00'
AND date <= '2010-12-31 23:59:59'
GROUP BY product) C
USING (product)
(SELECT product,COUNT(1) cnt FROM stats_entry
WHERE date >= '2011-01-01 00:00:00'
AND date <= '2011-12-31 23:59:59'
GROUP BY product) D
USING (product);

Give it a Try !!!