Question

Background

I run (read: inherited) a network that is setup very similar to a shared hosting provider. There are between 300-400 sites running on the infrastructure. Over the years the database topology has become extremely fragmented, in that it’s a 1 to 1 relationship from webserver->database.

Problems

• The applications are 9 times out of 10 designed by third party design firms that have implemented wordpress/joomla/drupal etc.
• The databases are sort of haphazardly spread across 6 database servers. They are not replicated anywhere.
• The applications have no concept of separate database handles to separate INSERT to a master and a SELECT to a slave.
• Using single master builtin mysql replication creates a huge bottleneck. The amount of inserts will down the master db very quickly.

Question

My question becomes, how can I make my database topology as flat as possible while leaving room for future scalability?

In the future I’d like to add more geographic locations to my network that can replicate the same databases across a ‘backnet’.

In the past I’ve looked into multi-master replication but saw a lot of issues with things like auto_increment column collisions.

I’m open to enterprise solutions. Something similar to the Shareplex product for Oracle replication.

Whatever the solution is, it’s not reasonable to expect the applications to change to accommodate this new design. So things like auto_increment columns need to remain the same and gel across the entire cluster.

Goal

My goal is to have an internally load balanced hostname for each cluster that I can point all the applications at. I

This would also afford me fault tolerance that I don’t presently have. At present, removing a database from rotation is not possible.

Applications like Cassandra and Hadoop look amazingly similar to what I want to achieve but NoSQL isn’t an option for these applications.

Any tips/pointers/tutorials/documentation/product recommendations are greatly appreciated. Thank you.

Answer 1

In the past I’ve looked into multi-master replication but saw a lot of issues with things like auto_increment column collisions.

We use multi-master in production at work. The auto-inc conundrum was fixed a while ago with auto_increment_increment and auto_increment_offset, which allows each server to have it’s own pattern of increment IDs. As long as the application isn’t designed blindly assuming that all the IDs will be sequential, it should work fine.

The real problem with multi-master is that MySQL still occasionally corrupts the binary log. This is mainly a problem over unreliable connections, so it won’t be a problem if all the instances are local.

Another problem with multi-master is that it simply doesn’t scale with writes, as you’ve already experienced or assumed, given a point in your answer. All of the writes on one master have to be replicated by the others. Even if you spread out read load correctly, you will eventually hit an I/O bottleneck that can only be resolved by either more hardware, an application redesign, or sharding (read: application redesign). It’s slightly better now that MySQL has row-based replication available.

If you need geographic diversity, multi-master could work.

Also look into DRBD a disk-block-level replication system that’s now been built into modern Linux kernels. It’s been used by others to replicate MySQL and PostgreSQL before, though I don’t have any personal experience with it.

I can’t tell from your question if you’re looking for high availability, or simply are looking for (manual or automatic) fail-over. If you just need fail-over and can have a bit of downtime, then traditional master/slave replication might be enough for you. The trouble is turning a slave that became a master back into a slave.