Question

I’d like to get feedback on how to model the following:

• Two main objects: collections and resources.
• Each user has multiple collections. I’m not saving user information per se: every collection has a “user ID” field.
• Each collection comprises multiple resources.
• Any given collection belongs to only one user.
• Any given resource may be associated with multiple collections.

I’m committed to using MySQL for the time being, though there is the possibility of migrating to a different database down the road. My main concern is scalability with the following assumptions:

• The number of users is about 200 and will grow.
• On average, each user has five collections.
• About 30,000 new distinct resources are “consumed” daily: when a resource is consumed, the application associates that resource to every collection that is relevant to that resource. Assume that typically a resource is relevant to about half of the collections, so that’s 30,000 x (1,000 / 2) = 15,000,000 inserts a day.
• The collection and resource objects are both composed of about a half-dozen fields, some of which may reach lengths of 100 characters.
• Every user has continual polling set up to periodically retrieve their collections and associated resources–assume that this happens once a minute.

Please keep in mind that I’m using MySQL. Given the expected volume of data, how normalized should the data model be? Would it make sense to store this data in a flat table? What kind of sharding approach would be appropriate? Would MySQL’s NDB clustering solution fit this use case?

Answer 1

Given the expected volume of data, how normalized should the data model be?

Perfectly.

Your volumes are small. You’re doing 10,000 to 355,000 transactions each day? Let’s assume your peak usage is a 12-hour window. That’s .23/sec up to 8/sec. Until you get to rates like 30/sec (over 1 million rows on a 12-hour period), you’ve got get little to worry about.

Would it make sense to store this data in a flat table?

No.

What kind of sharding approach would be appropriate?

Doesn’t matter. Pick any one that makes you happy.

You’ll need to test these empirically. Build a realistic volume of fake data. Write some benchmark transactions. Run under load to benchmarking sharding alternatives.

Would MySQL’s NDB clustering solution fit this use case?

It’s doubtful. You can often create a large-enough single server to handle this load.

This doesn’t sound anything like any of the requirements of your problem.

MySQL Cluster is designed not to have any single point of failure. In
a shared-nothing system, each component is expected to have its own
memory and disk, and the use of shared storage mechanisms such as
network shares, network file systems, and SANs is not recommended or
supported.