Question

I often write little Python scripts to iterate through all rows of a DB-table.
For example sending all to all subscribers a email.

I do it like this

conn = MySQLdb.connect(host = hst, user = usr, passwd = pw, db = db)
cursor = conn.cursor()
subscribers = cursor.execute("SELECT * FROM tbl_subscriber;")

for subscriber in subscribers:
 ...

conn.close()

I wonder if there is a better way to do this cause it is possible that my code loads thousands of rows into the memory.

I thought about that it could be done better with LIMIT.
Maybe something like that:

"SELECT * FROM tbl_subscriber LIMIT %d,%d;" % (actualLimit,steps)    

Whats the best way to do it?
How would you do it?

Answer 1

unless you have BLOBs in there, thousands of rows shouldn’t be a problem. Do you know that it is?

Also, why bring shame on yourself and your entire family by doing something like

"SELECT * FROM tbl_subscriber LIMIT %d,%d;" % (actualLimit,steps)

when the cursor will make the substitution for you in a manner that avoids SQL injection?

c.execute("SELECT * FROM tbl_subscriber LIMIT %i,%i;", (actualLimit,steps))