Question

We run a MySQL server with moderate load (200-300 QPS) on quite powerful hardware (HP DL360 with 8 Xeon cores, 8Gb RAM and RAID10). All the tables are innodb and the active dataset fits within the allocated innodb_buffer_pool_size.

Our database is normalized and to reduce the number of joins we use materialized views to flatten the dataset. As the data is added in batches a few times a day the MV:s are regenerated using CREATE TABLE AS SELECT instead of dynamically updated using complex triggers.

The problem is that sometimes while these CREATE queries are run (each of which takes anything from 5 to 50 seconds) other unrelated queries to the server seems to get queued up behind the CREATE query, leading to a unresponsive database.

To (re-)generate the MV:s we use something like this:

BEGIN TRANSACTION;
DROP TABLE IF EXISTS TableName_TMP;
CREATE TABLE TableName_TMP ENGINE=INNODB CHARACTER SET utf8 COLLATE utf8_swedish_ci AS 
    SELECT about100columns, and10Expressions 
    FROM Table1 
    JOIN Table2 ON Table1.fk = Table2.pk 
    /* join up to 13 other tables */
    WHERE ((removed IS NULL OR removed = 0)) 
    ORDER BY created DESC, id ASC;
ALTER TABLE TableName_TMP ADD PRIMARY KEY(id), INDEX(created);
DROP TABLE IF EXISTS TableName;
ALTER TABLE TableName_TMP RENAME TO TableName;
COMMIT;

The EXPLAIN of the SELECT produces something like:

+----+-------------+------------------+-------------+---------------+------------+---------+------------------------------+-------+-----------------------------+
| id | select_type | table            | type        | possible_keys | key        | key_len | ref                          | rows  | Extra                       |
+----+-------------+------------------+-------------+---------------+------------+---------+    ------------------------------+-------+-----------------------------+
|  1 | SIMPLE      | Table1           | ref_or_null | removed       | removed    | 5       | const                        | 76093 | Using where; Using filesort | 
|  1 | SIMPLE      | Table2           | eq_ref      | PRIMARY       | PRIMARY    | 4       | Table1.fk1                   |     1 |                             | 
|  1 | SIMPLE      | Table3           | eq_ref      | PRIMARY       | PRIMARY    | 4       | Table1.fk2                   |     1 |                             | 
/* More of the same */
|  1 | SIMPLE      | TableN           | eq_ref      | PRIMARY       | PRIMARY    | 4        | TableM.fk                    |     1 | Using index                 | 
|  1 | SIMPLE      | TableX           | eq_ref      | PRIMARY       | PRIMARY    | 4       | TableY.fk                    |     1 |                             | 
/* More of the same */    
+----+-------------+------------------+-------------+---------------+------------+---------+------------------------------+-------+-----------------------------+

Any ideas why the CREATE TABLE AS completly overload our server and how I can prevent it?

Regards,

Answer 1

We solved this by switching to SELECT INTO and LOAD DATA INFILE as by http://www.mysqlperformanceblog.com/2006/07/12/insert-into-select-performance-with-innodb-tables/ . Many thanks to Randolph Potter for sending us in the right direction.