Question

I’m working with a database schema that is running into scalability issues. One of the tables in the schema has grown to around 10 million rows, and I am exploring sharding and partitioning options to allow this schema to scale to much larger datasets (say, 1 billion to 100 billion rows). Our application must also be deployable onto several database products, including but not limited to Oracle, MS SQL Server, and MySQL.

This is a large problem in general, and I’d like to read up on what options are available. What resources are out there (books, whitepapers, web sites) for database sharding and partitioning strategies?

Answer 1

I agree with the other answers that you should look at your schema and indexes before resorting to sharding. 10 million rows is well within the capabilities of any of the major database engines.

However if you want some resources for learning about the subject of sharding then try these:

• Scalability Best Practices: Lessons from eBay
• Randy Shoup on eBay’s Architectural Principles – Video and Presentation
• High Scalability Site
• Mr. Moore gets to punt on sharding (when not to do it)