Question

Our server (several Java applications on Debian) handles incoming data (GNSS observations) that should be:

1. immediately (delay <200ms) delivered to other applications,
2. stored for further use.

Sometimes (several times a day maybe) about million of archived records will be fetched from the database. Record size is about 12 double precision fields + timestamp and some ids. There are no UPDATEs; DELETEs are very rare but massive. Incoming flow is up to hundred records per second. So I had to choose storage engine for this data.

I tried using MySQL (InnoDB). One application inserts, others constantly check last record id and if it is updated, fetch new records. This part works fine. But I’ve met following issues:

1. Records are quite large (about 200-240 bytes per record).
2. Fetching million of archived records is unacceptable slow (tens of minutes or more).

File-based storage will work just fine (since there are no inserts in the middle of DB and selections are mostly like ‘WHERE ID=1 AND TIME BETWEEN 2000 AND 3000’, but there are other problems:

1. Looking for new data might be not so easy.
2. Other data like logs and configs are stored in same database and I prefer to have one database for everything.

Can you advice some suitable database engine (SQL preferred, but not necessary)? Maybe it is possible to fine-tune MySQL to reduce record size and fetch time for continious strips of data?

MongoDB is not acceptable since DB size is limited on 32-bit machines. Any engine that does not provide quick access for recently inserted data is not acceptable too.

Answer 1

There really is no getting around how long it takes to load millions of records from disk. Your 32-bit requirement means you are limited in how much RAM you can use for memory based data structures. But, if you want to use MySQL, you may be able to get good performance using multiple table types.

If you need really fast non-blocking inserts. You can use the black hole table type and replication. The server where the inserts occur has a black hole table type that replicates to another server where the table is Innodb or MyISAM.

Since you don’t do UPDATEs, I think MyISAM would be better than Innodb in this scenario. You can use the MERGE table type for MyISAM (not available for Innodb). Not sure what your data set is like, but you could have 1 table per day (hour, week?), your MERGE table would then be a superset of those tables. Assuming you want to delete old data by day, just redeclare the MERGE table to not include the old tables. This action is instantaneous. Dropping old tables is also extremely fast.

To check for new data, you can look at “todays” table directly rather than going through the MERGE table.