Question

This piece of mysql code

SELECT  id, value, LENGTH(stuffing)
FROM  t_limit ORDER BY id LIMIT 150000, 10

can be optimized for better performance by rewriting it like this

Note:Table has Index on Id

SELECT  l.id, value, LENGTH(stuffing)
FROM    (
    SELECT  id
    FROM    t_limit
    ORDER BY
            id
    LIMIT 150000, 10
    ) o
JOIN    t_limit l
ON      l.id = o.id
ORDER BY
    l.id

Ref:http://explainextended.com/2009/10/23/mysql-order-by-limit-performance-late-row-lookups/

Now how to optimize this piece of code in a similar way

SELECT  id, value, LENGTH(stuffing)
FROM  t_limit where value>100 ORDER BY id LIMIT 150000, 10

Answer 1

The basic idea behind the optimization presented in the mentioned post is to query only the index pages without touching data pages. If you look at the query plan of the non-optimized query:

SELECT  id, value, LENGTH(stuffing) AS len
FROM    t_limit
ORDER BY
        id
LIMIT 150000, 10

it would be:

+----+-------------+---------+------+---------------+------+---------+------+--------+----------------+
| id | select_type | table   | type | possible_keys | key  | key_len | ref  | rows   | Extra
+----+-------------+---------+------+---------------+------+---------+------+--------+----------------+
|  1 | SIMPLE      | t_limit | ALL  | NULL          | NULL | NULL    | NULL | 200000 | Using filesort |
+----+-------------+---------+------+---------------+------+---------+------+--------+----------------+

So it’s a simple table-scan. With the subquery optimization we receive:

+----+-------------+------------+--------+-------------------------------+---------+---------+------+--------+---------------------------------+
| id | select_type | table      | type   | possible_keys                 | key     | key_len | ref  | rows   | Extra                           |
+----+-------------+------------+--------+-------------------------------+---------+---------+------+--------+---------------------------------+
|  1 | PRIMARY     | <derived2> | ALL    | NULL                          | NULL    | NULL    | NULL |     10 | Using temporary; Using filesort |
|  1 | PRIMARY     | l          | eq_ref | PRIMARY                       | PRIMARY | 4       | o.id |      1 |                                 |
|  2 | DERIVED     | t_limit    | index  | NULL                          | PRIMARY | 4       | NULL | 150010 | Using index                     |
+----+-------------+------------+--------+-------------------------------+---------+---------+------+--------+---------------------------------+

Look at the key column which shows that the inner most statement used the PRIMARY index. I slightly modified your query so the value type is compatible:

SELECT  l.id, value, LENGTH(stuffing) AS len
FROM    (
        SELECT  id
        FROM    t_limit
        where value like 'Value 1%'
        ORDER BY
                id
        LIMIT 30000, 10
        ) o
JOIN    t_limit l
ON      l.id = o.id
ORDER BY
        l.id

You need to consider what the where condition does. If you placed it in the outer query you would filter only the 10 rows returned from the inner query – I suppose that it was not what you were asking for. Now in the presented case (the where condition in the inner statement) you will end up with a table scan as there is no index that could fulfill your query:

+----+-------------+------------+--------+---------------+---------+---------+------+--------+--------------------------------+
| id | select_type | table      | type   | possible_keys | key     | key_len | ref  | rows   | Extra                           |
+----+-------------+------------+--------+---------------+---------+---------+------+--------+---------------------------------+
|  1 | PRIMARY     | <derived2> | ALL    | NULL          | NULL    | NULL    | NULL |     10 | Using temporary; Using filesort |
|  1 | PRIMARY     | l          | eq_ref | PRIMARY       | PRIMARY | 4       | o.id |      1 |                                 |
|  2 | DERIVED     | t_limit    | ALL    | NULL          | NULL    | NULL    | NULL | 200000 | Using filesort                  |
+----+-------------+------------+--------+---------------+---------+---------+------+--------+---------------------------------+

To profit from the same optimization as presented in the blog post you would need an additional non-clustered index, eg.

create index NCIX_t_limit_id_value on t_limit(id, value)

Now, when you run the above query the plan would be:

+----+-------------+------------+--------+-------------------------------+-----------------------+---------+------+-------+---------------------------------+
| id | select_type | table      | type   | possible_keys                 | key                   | key_len | ref  | rows  | Extra                           |
+----+-------------+------------+--------+-------------------------------+-----------------------+---------+------+-------+---------------------------------+
|  1 | PRIMARY     | <derived2> | ALL    | NULL                          | NULL                  | NULL    | NULL |    10 | Using temporary; Using filesort |
|  1 | PRIMARY     | l          | eq_ref | PRIMARY,NCIX_t_limit_id_value | PRIMARY               | 4       | o.id |     1 |                                 |
|  2 | DERIVED     | t_limit    | index  | NULL                          | NCIX_t_limit_id_value | 66      | NULL | 30010 | Using where; Using index        |
+----+-------------+------------+--------+-------------------------------+-----------------------+---------+------+-------+---------------------------------+

And again we are only scanning index pages.