Question

I have a very simple query I’m executing with PyMySQL:

SELECT `id` FROM `records` ORDER BY `id` DESC

records has over 1.5 million rows in it. id is a primary key.

Is this a limitation of PyMySQL? Is there something else I should be using if I’m querying so many rows at once?

Answer 1

You could break the query into several smaler queries:

from math import ceil

batch_size = 1000

for start_at in range(int(ceil(total_rows / 1000 * 1.0))):
    sql = 'SELECT `id` from `RECORDS` ORDER BY `id` DESC LIMIT %i, %i'
    sql = sql % (start_at * batch_size, batch_size)
    # fetch rows