Question

I’m updating an sqlite database to have a new column bin that is filled with the numbers 0 to 100. I’ve written a simple script that illustrates some performance issues I’m having:

import sqlite3

conn = sqlite3.connect('partition_dev.db')
db_parts = conn.cursor()

i = 0
for row in db_parts.execute('''SELECT * from parts''').fetchall():
    db_parts.execute('''UPDATE parts set bin=? where entry=?''',(i,str(row[0])))
    conn.commit()
    i = i+1
    print i
    if i>100:
        i=0

This runs an exeedingly long time for a 65K row database. I’m new to working with sql, so I assume I’m doing something obviously sub-optimal, but I’m not sure what. How can I increase the performance of this update?

Answer 1

You are committing every single row. Don’t do that, commit when all rows have been updated.

For the i counter, you can use an infinite generator; the count_to function will generate the numbers 0 through to 100 for you, starting back at 0 endlessly:

def count_to(n):
    while True:
        for i in range(n):
            yield i

to100 = count_to(101)
for row in db_parts.execute('''SELECT * from parts''').fetchall():
    i = to100.next()
    db_parts.execute('''UPDATE parts set bin=? where entry=?''',
        (i, str(row[0]))
    )
    print i
conn.commit()