Question

Im trying to process data from a database table. Loading them all simultaneously will hog most of computer resource. Heres my current code

cursor.execute("SELECT * from sometable")
rs = cursor.fetchall()
.
# process 1: set operation
.
for a in rs:
    # process 2: put data in another db

Is there a way to reduce resource usage? Like getting rows bit by bit under a loop?

Answer 1

You can use cursor.fetchone(). Read about it here. Example usage:

cursor.execute(query)
numrows = int(cursor.rowcount)
for x in range(0,numrows):
    row = cursor.fetchone()

You can also consider using LIMIT in the mysql query:

cursor.execute("SELECT * from sometable LIMIT 0, 100")

Finally, avoid using the star operator and only select the columns you need.