Question

One of our needs is to create an temporal in-memory database, to then perform various inserts/selects/updates.

At a glance SQLite satisfied all our needs. Connection to an in-memory SQLite DB can be established as simple as:

class SQLiteBase < ActiveRecord::Base
  self.abstract_class = true

  establish_connection(adapter:  'sqlite3', database: ':memory:')
end

A while ago we’ve started looking into some performace issues when it turned out that we need to perform bulk-loading (specifically, bulk INSERTs) of data into our SQLite tables (see these benchmarks).

Unfortunately, it looks like SQLite doesn’t support bulk INSERTs.

So is there any other SQL-based lightweight in-memory databases out there, that support bulk INSERTs?

If there are no such – is there a way to utilize heavyweight databases such as PostreSQL (MySQL or any other major player) as in-memory database?

If postresql/mysql is not a way-to-go – are there any other C-heavily-optimized data structures with a query language on top of it? (with/without a ruby binding).

Answer 1

First of all, SQLite does support bulk inserts using following methods:

• Poor man’s replacement for multi-valued INSERT:

  INSERT INTO mytable (a,b,c) SELECT 1 a, 2 b, 'x' c 
                    UNION ALL SELECT 2,   5,   'y' 
                    UNION ALL SELECT 3,   7,   'z'
                ...
  

• True multi-valued INSERT, supported since SQLite 3.7.11:

  INSERT INTO mytable (a,b,c) VALUES (1,2,'x'),
                                     (2,5,'y'),
                                     (3,7,'z');
  

• Using transactions:

  BEGIN;
  INSERT INTO mytable (a,b,c) VALUES (1,2,'x');
  INSERT INTO mytable (a,b,c) VALUES (2,5,'y');
  INSERT INTO mytable (a,b,c) VALUES (3,7,'z');
  COMMIT;
  

If you ask how to translate these into Ruby or Ruby on Rails – I admit, I have no idea, but I guess it should be possible.

But, even if SQLite did not support these methods, for in-memory database nothing of this should really matter – because if it is really all in-memory, insert speed should not really depend whether you insert rows one by one or as one transaction. Your speed limit is really just raw memory copy bandwidth.