Question

Designing an object store for ~10B objects and using mySQL for storing metadata. We know we need to shard and are looking for general heuristics to scale. Would appreciate pointers and hear about the back-of-the-envelope calculations you all used

Answer 1

You don’t know you need to shard. Find out first.

In preference order:

• Use a single server more efficiently.
• Do functional partitioning (keep different parts of your app on separate servers)
• Do vertical partitioning (keep different columns separately)
• Do horizontal partitioning (i.e. “sharding”)

This is because of the complexity impact on your application. It is possible to have a table with 10B rows in one server quite easily and expect good performance; it very much depends on your use case – how big the rows are and how many queries / updates you’re doing.

Good design is always the key; good application design is at least as important as good database design.

Don’t assume. Test.

Buy lots of hardware for your performance lab. Don’t buy any for production (yet).