I’m working on a site that’s grown both in terms of user-base and functionality to the point where it’s becoming evident that some of the admin tasks should be separate from the public website. I was wondering what the best way to do this would be.
For example, the site has a large social component to it, and a public sales interface. But at the same time, there’s back office tasks, bulk upload processing, dashboards (with long running queries), and customer relations tools in the admin section that I would like to not be effected by spikes in public traffic (or effect the public-facing response time).
The site is running on a fairly standard Rails/MySQL/Linux stack, but I think this is more of an architecture problem than an implementation one: mainly, how does one keep the data and business logic in sync between these different applications?
Some strategies that I’m evaluating:
1) Create a slave database of the public facing database on another machine. Extract out all of the model and library code so that it can be shared between the applications. Create new controllers and views for the admin interfaces.
I have limited experience with replication and am not even sure that it’s supposed to be used this way (most of the time I’ve seen it, it’s been for scaling out the read capabilities of the same application, rather than having multiple different ones). I’m also worried about the potential for latency issues if the slave is not on the same network.
2) Create new more task/department-specific applications and use a message oriented middleware to integrate them. I read Enterprise Integration Patterns awhile back and they seemed to advocate this for distributed systems. (Alternatively, in some cases the basic Rails-style RESTful API functionality might suffice.) But, I have nightmares about data synchronization issues and the massive re-architecting that this would entail.
3) Some mixture of the two. For example, the only public information necessary for some of the back office tasks is a read-only completion time or status. Would it make sense to have that on a completely separate system and send the data to public? Meanwhile, the user/group admin functionality would be run on a separate system sharing the database? The downside is, this seems to keep many of the concerns I have with the first two, especially the re-architecting.
I’m sure the answers are going to be highly dependent on a site’s specific needs, but I’d love to hear success (or failure) stories.
As far as I can tell from your description, if you really want to make sure public facing performance and admin performance don’t impact each other, you will have to use separate databases on separate servers.
The key is to get a good understanding of what data is used where and how the data is involved in the process. If you can do that then try to determine if you can make one database the ‘leading’ one. This would be your live data and the other database would be your operation data store (ODS). The ODS is always a derivative from your live data. The process of updating the ODS from the live data can happen at an interval and can both simplify data and do additional processing to make it more appropriate for the application using the ODS.
Now if you find this will be to much of a leap for your current situation, you can try and at least seperate the data within the database, so you won’t be dealing with performance issues like table locks, etc.. It can also be a step in the right direction for if you need to move to an ODS like model in the future.