Question

I have an sqlite table with a few hundred million rows:

sqlite> create table t1(id INTEGER PRIMARY KEY,stuff TEXT );

I need to query this table by its integer primary key hundreds of millions of times. My code:

conn = sqlite3.connect('stuff.db')
with conn:
    cur = conn.cursor()
    for id in ids:
        try:
            cur.execute("select stuff from t1 where rowid=?",[id])
            stuff_tuple = cur.fetchone()
            #do something with the fetched row
        except:
            pass #for when id is not in t1's key set

Here, ids is a list that may have tens of thousands of elements. Forming t1 did not take very long (ie ~75K inserts per second). Querying t1 the way I’ve done it is unacceptably slow (ie ~1K queries in 10 seconds).

I am completely new to SQL. What am I doing wrong?

Answer 1

Since you’re retrieving values by their keys, it seems like a key/value store would be more appropriate in this case. Relational databases (Sqlite included) are definitely feature-rich, but you can’t beat the performance of a simple key/value store.

There are several to choose from:

• Redis: “advanced key-value store”, very fast, optimized for in-memory operation
• Cassandra: extremely high performance, scalable, used by multiple high-profile sites
• MongoDB: feature-rich, tries to be “middle ground” between relational and NoSQL (and they’ve started offering free online classes)

And there’s many, many more.