Question

I have a table with around 6 million rows and has many columns with lot of data.

I want to implement a search functionality on this table mostly based on date range when the row was created and some other columns.

No other table is required so no question of joins in query. Also, I have implemented indexes on date column and other relevant columns.

Still the search takes long time if I search for large ranges, considering number of rows the table has. I have increased key_buffer_size to 1G and also mysql_query_cache to 500M.

I want to know if,

1. There is anything else I can do, optimizing any other variables that may increase performance.
2. Should I look into re-designing the table itself?

I want to explore all options before increasing RAM or CPU.

UPDATE:

Below are more details

CPU: Intel Quadcore
RAM: 4GB
OS: CentOS

+-----------------------------+---------------+------+-----+---------+----------------+
| Field                       | Type          | Null | Key | Default | Extra          |
+-----------------------------+---------------+------+-----+---------+----------------+
| deal_id                     | int(11)       | NO   | PRI | NULL    | auto_increment | 
| deal_group_id               | int(11)       | NO   | PRI | NULL    |                | 
| site_id                     | int(11)       | YES  | MUL | NULL    |                | 
| datetime                    | char(14)      | YES  | MUL | NULL    |                |  
| deal_category               | varchar(200)  | YES  | MUL | NULL    |                | 
| vendor_name                 | varchar(200)  | YES  |     | NULL    |                | 
| area                        | varchar(200)  | YES  | MUL | NULL    |                | 
| address                     | text          | YES  |     | NULL    |                |  
| country_code                | varchar(5)    | NO   | MUL | NULL    |                | 
| expired                     | int(1)        | NO   |     | 0       |                | 
| type                        | varchar(20)   | NO   |     | REGULAR |                | 
| pn_id                       | int(11)       | NO   |     | 0       |                | 
+-----------------------------+---------------+------+-----+---------+----------------+

Above is the schema. In the where clause the combination of datetime in range, deal_category, vendor_name in like and country_code is used. All are indexed. Below is one sample query


SELECT distinct t.deal_id, t.deal_group_id, t.site_name, UNIX_TIMESTAMP(t.datetime) as datetime, t.deal_category as deal_category, t.vendor_name, t.area as area, t.country_code FROM table t WHERE 1 AND t.country_code = 'US' AND datetime BETWEEN '20110601' AND '20120303' limit 10;

I have increased innodb_buffer_pool_size to around 75%, it has improved the performance for some queries, but still takes lot of time if all filters are included.

Below is the my.cnf file settings,

query_cache_size=500M

max_allowed_packet= 1G

innodb_buffer_pool_size = 3G

key_buffer_size = 1G

This table is quite busy, there are hundreds of writes happening every minute.

Hope this gives clearer picture.

More Updates

Below is the explain output and indexes on table,

+----+-------------+-------+------+-----------------------+--------------+---------+-------+--------+-------------+
| id | select_type | table | type | possible_keys         | key          | key_len | ref   | rows   | Extra       |
+----+-------------+-------+------+-----------------------+--------------+---------+-------+--------+-------------+
|  1 | SIMPLE      | t     | ref  | country_code,datetime | country_code | 17      | const | 521556 | Using where | 
+----+-------------+-------+------+-----------------------+--------------+---------+-------+--------+-------------+

Indexes:

+-------------+------------+-------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
| Table       | Non_unique | Key_name          | Seq_in_index | Column_name   | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment |
+-------------+------------+-------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+
| table       |          0 | PRIMARY           |            1 | deal_id       | A         |     5810775 |     NULL | NULL   |      | BTREE      |         | 
| table       |          0 | PRIMARY           |            2 | deal_group_id | A         |     5810775 |     NULL | NULL   |      | BTREE      |         | 
| table       |          1 | site_id           |            1 | site_id       | A         |        2109 |     NULL | NULL   | YES  | BTREE      |        | 
| table       |          1 | area              |            1 | area          | A         |       18927 |     NULL | NULL   | YES  | BTREE      |         | 
| table       |          1 | country_code      |            1 | country_code  | A         |          20 |     NULL | NULL   |      | BTREE      |         | 
| table       |          1 | deal_group_id     |            1 | deal_group_id | A         |     1936925 |     NULL | NULL   |      | BTREE      |         | 
| table       |          1 | deal_category     |            1 | deal_category | A         |          20 |     NULL | NULL   | YES  | BTREE      |         | 
| table       |          1 | datetime          |            1 | datetime      | A         |     5810775 |     NULL | NULL   | YES  | BTREE      |         | 
+-------------+------------+-------------------+--------------+---------------+-----------+-------------+----------+--------+------+------------+---------+

Thanks

Answer 1

1. For a database running Innodb you need to increase the value of innodb_buffer_pool_size to about 70 – 80% available RAM on a dedicated server so that as much of the data as possible can fit in memory

2. Create a single index containing the fields that you are going to search on

3. Searches with LIKE %value% do not use indexes

For additional tuning you need to provide more information on the schema and searches you will be carrying out

1. Remove the ‘1 AND’ comparison in the WHERE clause

2. Change datetime BETWEEN ‘20110601’ AND ‘20120303’ to DATE(datetime) BETWEEN STR_TO_DATE(‘20110601’, ‘%Y%m%d’) AND STR_TO_DATE(‘20120303’, ‘%Y%m%d’). I am assuming that the datetime field contains dates in the MySQL format

3. Consider changing the datetime field from a char(14) to a datetime field for faster computations

4. If the country_code column only has 2 digit country codes consider changing its size to char(2)

5. Remove DISTINCT before deal_id column which is an autoincrement and primary key