Home/ Questions/Q 3501624
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-18T12:52:57+00:00

I have a python program in an embedded system that needs to write to

  • 0

I have a python program in an embedded system that needs to write to an sqlite database. This database is mission critical and thus needs to be fully synchronous. The problem is that the database commits take a LONG time (3-30 seconds) for a single insert. The insert is wrapped in a transaction, but there’s really no way to separate these inserts into multiple transactions.

I’ve been searching for any way to make the database commit take a shorter amount of time but I’m sort of lost.

I’ve tried setting pragma journal_mode=persistance, and that seemed to help but only a small amount. Now I’m thinking it could be sqlite being starved of i/o time.

Is there a way to increase JUST the process priority of sqlite3? I don’t want to increase the python priority itself because we’re doing logging, config updates, and other file i/o, but I want to force sqlite to take as much i/o time as possible.

I’m open to other suggestions to speed commit time up as well.

This is what I’m doing in my insert:

    cur = None
            try:
                logging.info('Inserting into : ' + table + ':' + str(m))
                sql = "INSERT INTO " + table + "("
                bind = " VALUES("
                list = [];
                for k, v in m.items():
                    if(v != None):
                        sql += k + ","
                        bind += "?,"
                        list.append(v)
                sql = str(sql).rstrip(',') + ")"
                bind = str(bind).rstrip(',') + ")"

                cur = conn.cursor()
                cur.execute("PRAGMA journal_mode=PERSIST")
                logging.info(sql + bind)
                cur.execute(sql + bind, list) # It's definitely this that takes the most time. Yes I've profiled.
                conn.commit()
                id = cur.lastrowid
                return id
            except Exception as e:
                raise e
            finally:
                if cur != None:
                    cur.close()
Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Have you tried 3.7’s WAL?

    Historically, SQLite has been very slow at handling safe writes (eg. without changing to synchronous=off). It wrote the entire transaction to a journal, flushed it to disk, then copied the whole thing back over the original file, with many blocking syncs happening in between, serializing the whole thing.

    SQLite 3.7’s write-ahead logging (WAL) should largely fix this problem; it avoids the redundant write, which is expensive for large transactions, and significantly reduces the number of required FS syncs.

    See: http://www.sqlite.org/wal.html

    • 0