Question

I’m looking for some advice regarding a multilingual MySQL Database Structure which can handle huge amounts of data.
We are using the following method at the moment:

Articles  <- Article_translations -> Languages
id           id                      id
date         language_id (fk)        locale
category     article_id  (fk)   
             content

Ok, lets just say we’ve got like 100.000 Articles and 5 languages…well..you see the problem. The larger the data, the slower the database (just a guess here, but complex JOIN queries which are absolutely necessary probably won’t be O(log(n)) but rather something like O(n^2)).

Our current solution is to split the Article_translations into [locale]_article_translations (e.g. en_us_article_translation) in which case we would need to synchronize the structure between those tables easily. Is this an appropriate method to solve this problem or are there better ones? If this is a good solution, is there something out there which could help to monitor changes (only structural, no data synch!) and synchronize those structures?

Answer 1

Assuming if you tune your query properly

• Check query execution plan with the large set of data
• Make sure if you use DB level parameter as “large set” instead of row level
• See if you make your table denormalized(or vise versa) enough.

I would suggest belows although I am not sure which version of MySQL your are using

• Partitioing at DB level
• Fast hard disk in the DB server

I would suggest to use partitioning first and then you might consider to upgrade hard disk.

Partitioning

Partitioning is data spliting provided by database level.
Based on your query usage, you can divide data, for example, by language in your case.
The good thing to use DB partitioning is that

• it could be treated by a single table from application side
• Depends on the data volumn and frequncy, it can be rearranged by DB level. No impact to the apps.

Hard disk quality

Also the hard disk quality is important to handle large set of data.
Even if the query is tunned at best, if you deal with lots of data in a single query, you need fast data access. But this is costy.