Question

I’m working on optimizing a MySQL query, but am running into a strange issue. I never work with the live production database, so I create a dump of it and import it into a database on my local machine using mysqldump with no extra options.

The versions of mysql on the production database and my local virtual machine are nearly exactly the same:

Production:
mysql Ver 14.14 Distrib 5.1.61, for debian-linux-gnu (x86_64) using readline 6.2

Virtual Machine:
mysql Ver 14.14 Distrib 5.1.63, for debian-linux-gnu (x86_64) using readline 6.2

This one query is pretty complicated and takes around 4-5 seconds on production, but takes less than 1 second on the VM. The only thing I can think of is that maybe there are locks on the production database that prevent the query from running right away, and the query must wait on the locks?

The EXTENDED EXPLAIN when running the query against each database is nearly the same, with a couple slight differences.

I’m using SQL_NO_CACHE before the query to ensure the queries are not hitting the cache.

So my questions are:

1. What could cause EXTENDED EXPLAIN to differ, even slightly, when I’m working with a copy of the production database, and the mysql versions are the same?
2. Is there any reason that I’m not thinking of that would cause the same query to take longer on the production database?

Answer 1

There’s a number of things that could affect this execution time:

What is the loading like on your production system? How much IO headroom do you have? If your system is busy running other queries or doing heavy IO, the performance of any query will suffer severely.

Generally you can use SHOW PROCESSLIST to help identify what’s currently running and a program like iotop to see how much IO is going on.

Is your virtual machine running off of SSD? Even a mid-range development machine with SSD will blow away nearly any HD based database server for anything that’s heavily dependent on random access.

Have you ever tried OPTIMIZE TABLE on the production system? When you do a restore the table is always optimized automatically. On a live system the table will slowly degrade as you INSERT and DELETE rows.

Make sure both are using the same storage engine. MyISAM is often faster than InnoDB but isn’t safe to use in a production environment for critical data. Check with SHOW TABLE STATUS to see what engine is used for each table. You may have different defaults.

Also check that /etc/my.cnf is tuned accordingly. The default configuration for MySQL is terrible. You really need to allocate more memory for InnoDB buffers or performance will be awful.