Question

Lately my queries before caching into memcache have been taking forever to process! In this example, it took 10 seconds. All I am trying to do is get the 10 most recent hits in this case.

I am getting the feeling that it loads all 125,592 rows then only returns 10, am I right?

# User@Host: root[root] @ localhost [] # Query_time: 10  Lock_time: 0  Rows_sent: 10  Rows_examined: 125592 SELECT * FROM hits WHERE campaign_id = 30 ORDER BY id DESC LIMIT 10;

Here is another slow query:

 # Time: 090214  5:00:40 # User@Host: root[root] @ localhost [] # Query_time: 3  Lock_time: 0  Rows_sent: 1  Rows_examined: 128879 SELECT count(DISTINCT(ip_address)) AS count_distinct_ip_address FROM `hits` WHERE (campaign_id = 30);

When running the query the phpMyAdmin, it takes 1.3395 seconds. Although just doing a SELECT * FROM hits only takes 0.0001 seconds. I find it very odd that returning all of the hits takes less then sorting through them, or is it just that, I am sorting through them?

For those who want to see my table:

CREATE TABLE `hits` (   `id` int(11) unsigned NOT NULL auto_increment,   `hostname` varchar(255) NOT NULL,   `url` tinytext NOT NULL,   `user_agent` tinytext NOT NULL,   `created_at` timestamp NOT NULL default CURRENT_TIMESTAMP,   `ip_address` varchar(15) NOT NULL,   `campaign_id` int(11) NOT NULL,   PRIMARY KEY  (`id`),   KEY `campaign_id` (`campaign_id`),   KEY `ip_address` (`ip_address`) );

Answer 1

It seems your campaign_id index has low selectivity, i. e. there are lots of records with this value.

Ordering so many record takes a lot of time.

Try to use INDEX SCAN on the PRIMARY KEY for ordering:

/* Edited, as MySQL does not use live feed from the derived source with ORDER BY */ SELECT * FROM hits WHERE IFNULL(campaign_id, campaing_id) = 30 ORDER BY id DESC LIMIT 10; 

As for your second query, there is not much that may be done, as you need a complete scan on whole campaign_id = 30 anyway, be it TABLE SCAN or INDEX SCAN.

In fact, the TABLE SCAN can be even faster:

SELECT count(DISTINCT(ip_address)) AS count_distinct_ip_address FROM `hits` WHERE IFNULL(campaign_id, campaign_id)  = 30; 

If it is not, you may create an index on (campaign_id, ip_address) and use a trick to imitate INDEX GROUP BY on this index:

CREATE INDEX ix_hits_campaign_ip ON hits(campaign_id, ip_address)  SELECT SUM(cnt) FROM ( SELECT CASE WHEN @r = ip_address THEN 0 ELSE 1 END AS cnt,   @r := ip_address FROM   (SELECT @r:='') r,   (   SELECT ip_address   FROM hits   WHERE campaign_id = 30   ORDER BY ip_address   ) i ) o 

The trick here is simple: we don’t need the result, just a count, so there is no need in scanning for actual values. Index scan will suffice.

Unfortunately, despite what MySQL documentation says here on loose index scans, they do not actually work on composite indices. That’s why we need to imitate an INDEX SCAN WITH GROUP BY.

We do it by forcing MySQL to use INDEX RANGE SCAN that retrieves all records with campaign_id = 30 sorted by ip_address. Then we count DISTINCT ip_address‘es using a session variable @r initialized to an empty string in the first subquery.

In the first field we set the variable to 0 when the previous ip_address (stored in the variable) equals to the current one; otherwise we set it to 1. In the second field we assign the current value of ip_address to the variable.

Finally we retrieve the SUM on the first field which will of course give us COUNT (DISTINCT ip_address).