So, I am currently rebuilding my web platform from a single-machine to a cluster of machines, and I will be using Apache load balancing to do this., but I have two questions that I need a good answer to before proceeding. I have Googled and searched here in SO, but didn’t find anything good.
My setup will be one Debian machine running the Apache load balancing server (i.e. Apache with mod_proxy) and then any number of “slave” machines, that are balancing members. All of these are VPS inside a VMWare machine, so setting up new slaves as needed will be trivial.
Log Files
The first question is that of log files. In order to troubleshoot my platform, I sometimes need to analyze log files, both access logs and error logs, from Apache. When the load is evenly distributed (i.e. I don’t know if I’ll even use sticky balancing, any host could probably handle any request at any time), so would the log files for each slave Apache instance. Is there a way to consolidate these live, meaning that my live log analyzer could see the log files from all hosts? I certainly understand that doing so while the files being on several hosts would be difficult, so is there a way to make sure that all log files are kept on one server?
I’m thinking about two things myself, but I would greatly appreciate your input.
syslogd
The first is syslogd, where in it would be possible for several hosts to write to one logging host. The problem with this is that in my current setup, each virtual host in apache, has its own log file. That could probably be fixed in some manner though. My main usage for this is for troubleshooting, not keeping separate logs for each host (albeit if both goals could be met, that would certainly be a bonus).
NFS
My next thought was about NFS, i.e. having a NFS share on the LAN where each slave can write to the same log file. I’m going to go ahead and assume that this will be difficult since slave 1 would open the log file and then slave 2 wouldn’t be able to write to it.
As I said, your input is greatly appreciated since I feel I’m stuck in how to solve this.
Configuration files
This is another thing altogether. Each slave will respond to each request as if acting as one single server. That is the entire idea. But what about making changes to the apache configuration files, adding virtual hosts, setting up other parameters? What if I have ten slaves, or fifty? Is there a way to make sure that all these slaves are always in sync? I am already using a NFS export to make sure they all have the same files, but should I use the same approach with the configuration files? Or should I have these as some form of repository and then use rsync to copy them out to the slaves? One problem is that I have built an interface in my web platform that edits these configuration files (namely the file with the virtual hosts), and since that action would take place on one of the slaves, the most current copy of this file could potentially be on one slave.
I realize that this was a long and wieldy post, and I apologize. I just wanted to make sure that all the parameters of my problem were expressed.
I hope someone out there can help me, as you have before! Thank you in advance!
NFS will not help you with log files, for exactly the reasons you describe above. You should use syslogd (or some other solution like Splunk) to centralize the logging. It’s trivial to include information about what host the log entry comes from, so you can still winnow down to per-host data when troubleshooting.
Configuration files: you need to either centralize them (a “master” copy), or have a way of distributing changes made on any server to all the others. I recommend centralization as the simpler approach. NFS will do the job here, or, as you suggest, a repository from which all hosts are periodically updated. There are a lot of options here, running all the way up to version control (SVN, git, etc) or even configuration servers (Chef, etc).
Please note that moving from a single server to a cluster has many implications. In both cases above (logging, config files), there is potential to introduce single points of failure if done naively. Since you have that already (one server), you’re not worse off, but you should try to be aware of and plan for the failure scenarios you may need to respond to.