In order to improve insert performance and load on server, I’ve decided to divide a large table into 2. A big table which will only be used for “select” and a smaller table which will be used mainly for “insert” and sometimes also “select”. Each time period (I thought about a day) I will merge the smaller table into the big one.
Regrading the big table: is there a way I can improve performance by telling the mysql server it’s read-only? Considering it’s only for select, can I assume it will handle SELECT in less than 1sec. when it will become ~1e9 rows?
Regarding the small table: any tuning I should do here? what is the best way to develop an automated merge process from the small to the big table (in php)?
I’ve seen this and similar strategies used for this purpose and it has always been counterproductive. You end up creating complexity with more overhead that you gain. For instance, you are guaranteeing that every record gets inserted at least twice.
Be sure to benchmark carefully and compare to confirm your hypothesis before going too far with this.
I’ve found it’s highly effective to circumvent deletes on MyISAM tables. You could for instance switch to a new small table once a day and merge the old one into the big one once both are read-only; then drop the old one. Once deletes start showing up in a MyISAM table its efficiency drops significantly from having to find and reuse empty holes.
Konerak’s suggestion about packed ISAM tables is a good complement to this point of view. One effect is that it gives you a table (and indexes) without holes.