Question

I’m using Python 3.2.1 on Arch Linux x86_64.
I’m trying to update an sqlite database in a threaded, timed loop with some code similar to the following:

import sqlite3
from threading import Timer
from queue import Queue

class DBQueue(Queue):
    def give(self, item):
        self.task_done()
        self.join()
        self.put(item)
        return True


def timer():
    print('A')
    Timer(3, add).start()


def add():
    print('B')
    db = qdb.get()
    cur = db.cursor()
    cur.execute('INSERT INTO Foo (id) VALUES (NULL)')
    qdb.give(db)
    timer()

qdb = DBQueue()
# SOLUTION #1:
# qdb.put(sqlite3.connect(':memory:', check_same_thread=False))
# SOLUTION #2: see Eli Bendersky's answer
qdb.put(sqlite3.connect(':memory:'))
db = qdb.get()
cur = db.cursor()
cur.execute('CREATE TABLE Foo (id INTEGER PRIMARY KEY)')
qdb.give(db)
timer()

which unfortunately returns:

A
B
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.2/threading.py", line 736, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.2/threading.py", line 942, in run
    self.function(*self.args, **self.kwargs)
  File "/home/dario/dev/python/prova/src/prova4.py", line 27, in add
    cursor = db.cursor()
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread.The object was created in thread id 140037302638336 and this is thread id 140037262886656

Sharing only a cursor doesn’t give better results:

conn = sqlite3.connect(':memory:')
qdb.put(conn.cursor())

I’m quite sure I haven’t understood at all how to use queues to share databases among threads, can anybody help me?
Thank you!

Answer 1

You don’t need the Queue – just use separate connections to the same database from the two threads. Keep in mind that you shouldn’t expect much in terms of ordering when separate connections commit data into the DB. Treat it as if you had two different instances of your program accessing the DB simultaneously.

---

If for some reason you feel that you absolutely must share a connection, then try this: since you hit a problem of creating SQLite objects from one thread and using them in another, why not delegate the task of handling the db/connection in a single thread, and let it communicate with others via Queues. More specifically:

• Thread DB_thread: “owns” the connections. Gets commands in a queue from other threads, executes them, places results in a “results queue”
• Threads A, B, C: pass “commands” into the queue and take results from the “results queue”.

Note that these commands are not SQLite objects.