We’re using MySQL and have an ~1.1 million record InnoDB table where the entire table has to be replaced with data from a CSV twice per day.
What I’ve been doing up until now is truncating the table, then using mysqlimport to repopulate the table from the fresh CSV twice a day. This results in around 160 seconds of downtime, which I was initially told was fine, but that is no longer the case.
The only way I can think of to do this is to have a staging table and a production table. Live queries will go to production, and when I need to repopulate the db, I’ll truncate staging, then mysqlimport the CSV into it, then after that is done I will swap the names so that staging will have the old database and production will contain the freshly imported CSV. I guess I would need an intermediate name, like production becomes production2, the staging becomes production, and then production2 becomes staging.
Does anyone have an alternate solution? The table needs to be imported as quickly as possible which is why I’m using mysqlimport, and the downtime must be extremely minimal.
I think you already have the optimal solution. Just make sure you do an atomic rename, i.e. “rename table production to production2, staging to production”. This way it will be transparent to the application/users. One thing to watch out for is if there are foreign keys that reference this table, because the keys will still reference the renamed old table. But I guess since you are able to truncate it before without any issue, you don’t have any in place.
EDIT: I’m quite new to commenting on StackOverflow so I don’t know if this is the best place to do so, but I just want to comment on the answer posted by eggyal – TRUNCATE causes an implicit commit so the proposed solution won’t work as described (i.e. the changes will be seen to the user/application even before the new data is loaded).