Question

I’ve built a site that has about 1,000,000 products in it. I want to build a set of site maps which will contain links to all the products. I understand that site maps may contain up to 50,000, links, so my plan is to build 20 such site maps.

I want to do accomplish this with a series of SQL statements. My plan: 1,000 SQL statements, each of which would pull up 1,000 products. The SQL statements would be along the following lines:

SELECT name, category, catalog_number FROM product ORDER BY id LIMIT    0, 1000
SELECT name, category, catalog_number FROM product ORDER BY id LIMIT 1000, 1000
SELECT name, category, catalog_number FROM product ORDER BY id LIMIT 2000, 1000
SELECT name, category, catalog_number FROM product ORDER BY id LIMIT 3000, 1000

And so on.

The problem is that as the first argument to LIMIT (the offset) grows, the amount of time each sql takes to execute grows very quickly! To be specific, when the first argument to LIMIT was anything less than 28000, the statement executed in under .3 seconds. But when I tried:

SELECT name, category, catalog_number FROM product ORDER BY id LIMIT 29000, 1000

the time taken to execute the statement jumped to 30 seconds! And every statement after that took a similarly long amount of time to execute.

Another thing I tried was to increase the second argument to LIMIT — but the same problem occurred.

Any assistance would be welcome. Thank you.

Answer 1

Using LIMIT requires MySQL to fetch all of the rows up to the limit, but discard those in which you’re not interested – i.e. merely don’t return them to the client. Therefore, if there are thousands of rows to fetch (and discard), it can dramatically impair the execution time.

Instead, just keep track of the last id you saw; then always filter on that:

SELECT   name, category, catalog_number
FROM     product
WHERE    id >= ?  -- the last one you saw, or 0/omit if none
ORDER BY id
LIMIT    1000

This will be especially fast if there is an index on the id column.