Question

Although I currently do not have it, I’m interested in learning how someone would scale an individual table in MySQL that might have, say 20 million users. Is this something you would use sharding for? What are some strategies one might use to make an individual table of this magnitude “scalable” ?

Answer 1

20M records is generally considered “small”. Depending on the size of records and the kind of queries performed, you are likely to get very good performance on the lowliest of servers.

Almost all servers can keep such a database in memory. Let’s consider that a record takes 1024 bytes, including indexes. This is quite a large record, yet 20M rows is still only 20Gb, which fits comfortably within the RAM of a modest server.

While your database fits in RAM, queries are likely to be very fast.

But in any case, you need to consider what the access patterns are.

Do you have

• Very high write rates – more than 100 transactions per second?
• Lots of hard queries / reports?

If the answer to both of these is “no”, you probably need no special equipment at all.

Certainly you don’t want to shard. It’s complicated, it massively complicates your application, and will require a LOT of developer time which is better spent on features (which you can actually sell to customers)

In order to improve performance with big data, in approximate order of preference, you want to:

• Buy better hardware (within reason)
• Reduce the amount of data you need to store
• Use horizontal partitioning
• Use vertical partitioning / functional partitioning
• Get a better database engine which can use existing hardware more efficiently (possible examples: Infobright, Tokutek)
• Shard (you really don’t want to do this!)