Question

I’m currently implementing some functionality relying on an SQLite database. While doing so, I’m being confronted with a limitation that I did not anticipate. It made me wonder whether I’m tackling my problem the right way. I’m hoping someone can suggest a different approach that takes into account the requirements.

Goal

The goal is to store a large collection of objects into a database as fast as possible. However, if at any point, it is determined that the storage of these objects should not take place, the operation should not have any effect on the database. The exact storage queries (the ‘inserts’) depend on previous inserts.

Use Case

Consider a very large collection of ‘instance information packages’, each package represents part of the knowledge of some corresponding abstract instance. The total collection thus represents the complete knowledge of one or more abstract instances. The information of each package consists of:

1. an instance_id; used to uniquely identify the abstract instance to which this information relates.
2. an attributes map; (attribute_uri, attribute_value)-pairs where attribute_uri uniquely identifies an attribute and attribute_value is a corresponding text value.

Both instance_id and attribute_id are found in a table, respectively TBL_INSTANCES and TBL_ATTRIBUTES. The database model used here is an EAV-model.

After insertion, a table TBL_ENTRIES should contain entries in the form:

[ entry_id | instance_id | attribute_id | attribute_value ]

Issue

The implementation seemed to be straightforward:

1. Set a savepoint
2. Begin the transaction
3. Prepare statement for many inserts in TBL_ENTRIES
4. For each instance in the collection:
   • 4.1. Lookup instance_id in TBL_INSTANCES. If not present, add it to TBL_INSTANCES
   • 4.2. For each attribute in the instance’s attribute map:
     • 4.2.1. Lookup attribute_id in TBL_ATTRIBUTES. If not present, add it to TBL_ATTRIBUTES.
     • 4.2.2. Add entry to TBL_ENTRIES
5. If at any point an error occured, rollback to the savepoint, otherwise commit.

Unfortunately, in the lookup steps (4.1 and 4.2.1), SQLite reports a “cannot start a transaction within a transaction” error. Reading the documentation, it seems to lie in the fact the lookups call the sqlite3_exec routines, which (unless I am mistaken) seems to setup a transaction internally.

Question

If nested transactions are not allowed, how am I supposed to perform the lookup steps (which are basically “INSERT OR REPLACE” expressions followed by a “SELECT” query)? Is there some way to avoid them or to use sqlite3_exec within the current transaction? If not, am I approaching my problem the wrong way? Perhaps I should not use separate integer as keys, e.g. use attribute_uri directly as key instead of looking up the corresponding attribute_id.

In any case, I still need to do something equivalent during the transaction as the real speed gain lies in preparing the statements over the collection of instances, not the collection of attributes: an instance might have just one or two attributes whereas the number of instances in the instance collection will always be very large.

I’m using C/C++, SQLite 3.7.

Any suggestions or remarks on my approach are much appreciated!

Answer 1

From SQLite documentation on BEGIN TRANSACTION:

Transactions created using BEGIN…COMMIT do not nest. For nested
transactions, use the SAVEPOINT and RELEASE commands

I seem to recall that nested transaction should be already there for any SQLite 3.7.*.