Question

We’re currently working on a python project that basically reads and writes M2M data into/from a SQLite database. This database consists of multiple tables, one of them storing current values coming from the cloud. This last table is worrying me a bit since it’s being written very often and the application runs on a flash drive.

I’ve read that virtual tables could be the solution. I’ve thought in converting the critical table into a virtual one and then link its contents to a real file (XML or JSON) stored in RAM (/tmp for example in Debian). I’ve been reading this article:
http://drdobbs.com/database/202802959?pgno=1
that explains more or less how to do what I want. It’s quite complex and I think that this is not very doable using Python. Maybe we need to develop our own sqlite extension, I don’t know…

Any idea about how to “place” our conflicting table in RAM whilst the rest of the database stays in FLASH? Any better/simpler approach about how take the virtual table way under Python?

Answer 1

A very simple, SQL-only solution to create a in-memory table is using SQLite’s ATTACH command with the special “:memory:” pseudo-filename:

ATTACH DATABASE ":memory:" AS memdb;
CREATE TABLE memdb.my_table (...);

Since the whole database “memdb” is kept in RAM, the data will be lost once you close the database connection, so you will have to take care of persistence by yourself.

One way to do it could be:

1. Open your main SQLite database file
2. Attach a in-memory secondary database
3. Duplicate your performance-critical table in the in-memory database
4. Run all queries on the duplicate table
5. Once done, write the in-memory table back to the original table (BEGIN; DELETE FROM real_table; INSERT INTO real_table SELECT * FROM memory_table;)

But the best advice I can give you: Make sure that you really have a performance problem, the simple solution could just as well be fast enough!