Question

I have 8 tables with over 2 millions rows using INT(4B) PK used for frequent inserting and reading. The older 9/10 of the data is read occasionally and doesn’t matter how long it takes to access it, while the newer 1/10 has to be fast for both INSERT and SELECT. The tables are divided into 2 categories of requirements:

1. 100 INSERTs/sec
2. 20 INSERTs/sec, occasional UPDATE

Because it should be working with innodb_buffer_pool_size set to 32M and the old data is unimportant, I think the best solution will be to say once per week to copy the older half of each table into big archive tables. Plus I should use infile inserting instead of the current transactions. Is it a good silution? I’ll appreciate any advice and link about this problem.

Answer 1

If you are using InnoDB, the data is naturally “clustered” using the PRIMARY KEY of the table, if you defined one (ie: “id INT NOT NULL PRIMARY KEY AUTO_INCREMENT”), the data is grouped by ID (and will stay that way).

So your most recent INSERTed data is naturally grouped on some InnoDB buffers, and your old archive data doesn’t matter at all.
I don’t think you will benefit from splitting data into archive tables/database and recent one, except you will make everything much more complex!

To speed-up insertion/updates/delete on InnoDB, you have to think about the physical location of your InnoDB log file: InnoDB needs to insert the modification into it to achieve the operation (wether it’s a explicit transaction or an implicit one!), it doesn’t wait for the data or index to be put back in disk. It’s a pretty different strategy than MyISAM.

So if you could allocate a fast sequential storage for the InnoDB log file, 10krpm+ hard-drive or SSD, and keep the ibdata in another drive or raid-array, you will be able to sustain an impressive number of DB modifications: it’s IO bound on InnoDB log file (exception is you use complex or heavy where clause on update/delete).