I am looking for some experiences from people that worked with big databases ( > 100GB ).
Basically I have a table that stores all messages sent by a system, for all accounts. Every week, we get aprox. +1 million new rows.
This table is starting to give problems with some queries. I understand how to proper optimize queries and tried different combinations of indexes. But i think that it’s just too much for one table.
One of the queries, which uses a range condition is what usually kills our database.
From your experience, is it better to split this table into several tables ( one for each account )? Each account already has its own database, so it’s not hard to modify our software for this.
I guess this would solve our problem with ranged queries, specially for small accounts with < 5k rows.
I haven’t worked with MySQL directly. I’m more of a SQL Server and Oracle guy. But I do know that MySQL has table partitioning, which breaks big tables and distributes them across files. You can set rules to the partitioning and how it all works.
Another tip you can try to do, is to separate your data into an archive table. For example, in your back end, if you need to search for messages by date and if the date range is over a certain amount of time, then your back end does its search in the archive table. Otherwise, it searches in the main table. This will allow you to maybe archive older messages.