Question

We are going to a deployment setup where we will have many servers, most of which auto-added to the load balancer when the traffic goes up. The trouble with this setup is that if an individual developer needs to tail logs to troubleshoot something, he will have to open a console on each server, which is made complicated by the fact that the developer often doesn’t know HOW many servers we might have in function at that time.

If the developer can find all the logs in one server – say our deploy server, then the troubleshooting becomes easier.

Towards this, I was thinking of setting up a push from each FE machine to the deploy server using a cron which will replicate the logs on our deploy server. There are two problems with this approach:

• There is a lag of 1 minute because crons can’t be run more frequently.
• The cron on each FE machine will have to set up to sync to a specific location on the deploy server, but up front, I don’t know how many such FE servers will exist.

To solve this problem, I am looking at a way to connect error_log, or PEAR Log to send the logs directly to our deploy server, which will log it in real time to it’s local locations at /var/log/..

Anybody knows how I can configure this? Or perhaps a service which accomplishes this?

Our servers are all Ubuntu 10.04 LTS and we are running these servers on EC2 instances in AWS. Our PHP version is 5.3.

Answer 1

The answer from @MichelFeldheim was the genesis, but we improved upon it to handle multiple applications writing to multiple log files.

---

Central Logging Server

In the central logging server, install syslog-ng and configure it thus:

sudo apt-get install syslog-ng

add the following to /etc/syslog-ng/syslog-ng.conf:

destination d_php { file("$PROGRAM" owner(www-data) group(www-data) perm(0644)); };
filter f_php { program("^\/var\/log\/"); };
log { source(s_all); filter(f_php); destination(d_php); flags(final); };

source s_all {
        # ....
        # .... LET THE PREVIOUS CONTENT STAY - add the following line
        tcp(port(5140) keep_alive(yes));
};

restart syslog service:

sudo service syslog-ng restart

---

On FE Servers

On each of the FE Servers, install syslog-ng and configure it thus:

sudo apt-get install syslog-ng

add the following to /etc/syslog-ng/syslog-ng.conf on each of the FE servers:

destination php { tcp("log.example.com" port(5140)); };
log { source(s_all); filter(f_php); destination(php); };
filter f_php { facility(user); };

restart syslog servers:

sudo service syslog-ng restart

---

Application Code Changes:

Now, the application code can be changed thus. Suppose each of the application have code like this writing to a separate file and you want the same structure to be reflected in the central log server:

// PREVIOUS CODE: using PEAR Log
include '/usr/share/php/Log.php';
$log = Log::singleton('file', '/var/log/nginx/xxx.log', '', array(), PEAR_LOG_INFO);
// PREVIOUS CODE: Using error_log
ini_set('error_log' , '/var/log/nginx/xxx.log');

The new code should look like:

// NEW CODE: using PEAR Log
include '/usr/share/php/Log.php';
$log = Log::singleton('syslog', LOG_USER, '/var/log/nginx/xxx.log', array(), PEAR_LOG_INFO);
// NEW CODE: Using error_log
ini_set(‘error_log’, ‘syslog’);
openlog('/var/log/nginx/xxx.log', LOG_NDELAY, LOG_USER);

If your FE servers and the Logging servers are all within the same EC2 security group, then there is no need to open the ports, since within the groups, all ports can be accessed freely, so long as a service is listening to it.

This approach allows your each of your apps, modules to decide whether they want central logging or not.