Question

I have a table with 3 million rows and 6 columns.

The table structure:

| Sample | CREATE TABLE `sample` (
  `ID` int(11) NOT NULL AUTO_INCREMENT,
  `FileMD5` varchar(32) NOT NULL,
  `NoCsumMD5` varchar(32) NOT NULL,
  `SectMD5` varchar(32) NOT NULL,
  `SectNoResMD5` varchar(32) NOT NULL,
  `ImpMD5` varchar(32) NOT NULL,
  `Overlay` tinyint(1) NOT NULL DEFAULT '1',
  PRIMARY KEY (`ID`),
  KEY `FileMD5` (`FileMD5`),
  KEY `NoCsumMD5` (`NoCsumMD5`)
) ENGINE=InnoDB AUTO_INCREMENT=3073630 DEFAULT CHARSET=latin1 |

The temporary table values:

mysql> SHOW VARIABLES LIKE 'tmp_table_size';
+----------------+----------+
| Variable_name  | Value    |
+----------------+----------+
| tmp_table_size | 16777216 |
+----------------+----------+
1 row in set (0.00 sec)
mysql> SHOW VARIABLES LIKE 'max_heap_table_size';
+---------------------+----------+
| Variable_name       | Value    |
+---------------------+----------+
| max_heap_table_size | 16777216 |
+---------------------+----------+
1 row in set (0.00 sec)

My Query

mysql> explain SELECT NoCsumMD5,Count(FileMD5) 
FROM Sample GROUP BY NoCsumMD5 
HAVING Count(FileMD5) > 10 ORDER BY Count(FileMD5) Desc ;
+----+-------------+--------+-------+---------------+-----------+---------+------+---------+---------------------------------+
| id | select_type | table  | type  | possible_keys | key       | key_len | ref  | rows    | Extra                           |
+----+-------------+--------+-------+---------------+-----------+---------+------+---------+---------------------------------+
|  1 | SIMPLE      | Sample | index | NULL          | NoCsumMD5 | 34      | NULL | 2928042 | Using temporary; Using filesort |
+----+-------------+--------+-------+---------------+-----------+---------+------+---------+---------------------------------+

How can I optimize this query. Even after 10 min, it generates no output.

I feel that I have indexed the right columns and given enough memory for temporary tables.

Answer 1

Since FileMD5 is not NULL in your table definition, the query can be simplified, and you will not need the composite index @brendan-long suggests (NoCsumMD5 index is enough):

SELECT NoCsumMD5, Count(*) as cnt 
FROM Sample
GROUP BY NoCsumMD5 
HAVING cnt > 10
ORDER BY cnt DESC;