Question

1. Below query is taking long time to execute around 2mins please help me how to improve performance of this query.
2. so our requirement is to get result within 2 to 3secs.
3. query is using indexes also.
4. but it was performing more scan.

query:

select max(`log_date`)
from `top_competitor_summary_entity`
where
  own_domain_id = 4
  and keyword_top1_count > 0
  and (grouptag_id = 0 OR grouptag_id is null);

Expalin Plan:

+----+-------------+-------------------------------+------+--------------------------------------+------------------------+---------+-------+---------+-------------+
| id | select_type | table                         | type | possible_keys                        | key                    | key_len | ref   | rows    | Extra       |
+----+-------------+-------------------------------+------+--------------------------------------+------------------------+---------+-------+---------+-------------+
|  1 | SIMPLE      | top_competitor_summary_entity | ref  | own_domain_id,own_domain_id_log_date | own_domain_id_log_date | 4       | const | 2100128 | Using where |
+----+-------------+-------------------------------+------+--------------------------------------+------------------------+---------+-------+---------+-------------+
1 row in set (0.66 sec)

Table structure:

mysql> show create table top_competitor_summary_entity\G
*************************** 1. row ***************************
       Table: top_competitor_summary_entity
Create Table: CREATE TABLE `top_competitor_summary_entity` (
  `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `domain` varchar(255) NOT NULL COMMENT 'competitor domain name',
  `own_domain_id` int(11) NOT NULL,
  `keyword_top10_count` int(11) DEFAULT NULL,
  `keyword_top3_count` int(11) DEFAULT NULL,
  `keyword_top1_count` int(11) DEFAULT NULL,
  `keyword_top10_search_volume` bigint(20) DEFAULT NULL,
  `keyword_top3_search_volume` bigint(20) DEFAULT NULL,
  `keyword_top1_search_volume` bigint(20) DEFAULT NULL,
  `url_top10_count` int(11) DEFAULT NULL
    COMMENT 'how many competitor url in Top 10',
  `log_date` date DEFAULT NULL,
  `grouptag_id` int(11) DEFAULT '0',
  `keyword_top10_count_bing` int(11) DEFAULT '0',
  `keyword_top10_count_yahoo` int(11) DEFAULT '0',
  `keyword_top3_count_bing` int(11) DEFAULT '0',
  `keyword_top3_count_yahoo` int(11) DEFAULT '0',
  `keyword_top1_count_bing` int(11) DEFAULT '0',
  `keyword_top1_count_yahoo` int(11) DEFAULT '0',
  PRIMARY KEY (`id`),
  KEY `own_domain_id` (`own_domain_id`),
  KEY `domain_own_domain_id_log_date` (`domain`,`own_domain_id`,`log_date`),
  KEY `own_domain_id_log_date` (`own_domain_id`,`log_date`)
) ENGINE=InnoDB AUTO_INCREMENT=680592051 DEFAULT CHARSET=utf8
1 row in set (0.09 sec)

Answer 1

The query itself is so simple that I don’t think there’s anything you could possibly do to it to make it faster by modifying it. Still I would be curious whether using “IsNull(grouptag_id, 0) = 0” would make any difference whatsoever. I doubt it, but it might be fun to see if it shaves anything off.

I think the real problem is that there are probably a ton of records that have an own_domain_id value of 4, and you don’t have indexes for the other fields in the where clause. You can create individual indexes for them and/or if you want to create an index that’s specifically tailored to this query then create one that keys on all 4 of the fields referenced.

Some other observations:

If it would be a possibility to change your code to deal with null values (perhaps just treat them as 0’s) then you could get rid of the default values you’re placing in most those fields and make them null instead. If not many fields actually have a value of 0 than this wouldn’t do much, but if a lot of fields are set to 0 than this would cause the table to take up less space on disk which would translate to less time to scan the table.

You could also partition the table either horizontally or vertically.

Horizontally: You could take all the top1 fields and put them in a top1 table, all the top3 fields and put them in a top3 table, etc. Alternatively, you might do all the yahoo in one table and all bing in another table; or perhaps all the count fields in one table and all the search fields in another. If you find yourself usually only need one set of fields at a time then this would reduce the search time, but if you usually end up grabbing all the fields in most your queries then of course it wouldn’t really help.

Vertically: This one is probably a lot more work than it’s worth, but you could split the records in your table over multiple tables and put them on multiple hard disks and query them all at the same time asynchronously. I’ve always wondered if Google does something along these lines.

I also notice you’re using a bigint for your id which is 8 bytes, as opposed to just an int which is only 4 bytes. If you think you will realistically be handling multiple billions of records at some point then bigint is obviously the way to go, but otherwise you could shrink your database by about a 100 megabytes which will also have the effect of making your searches marginally faster. And there’s no reason you can’t make it an int now and change it back to a bigint later if that becomes necessary.