Requirements
We have several servers (20-50) – Solaris 10 and Linux (SLES) – running a mix of different applications, each generating a bunch of log events into textfiles. We need to capture these to a separate monitoring box, where we can do analysis/reporting/alerts.
Current Approach
Currently, we use SSH with a remote “tail -f” to stream the logfiles from the servers onto the monitoring box. However, this is somewhat brittle.
New Approach
I’d like to replace this with RabbitMQ. The servers would publish their log events into this, and each monitoring script/app could then subscribe to the appropriate queue.
Ideally, we’d like the applications themselves to dump events directly into the RabbitMQ queue.
However, assuming that’s not an option in the short term (we may not have source for all the apps), we need a way to basically “tail -f” the logfiles from disk. I’m most comfortable in Python, so I was looking at a Pythonic way of doing that – the consensus seems to be to just use a loop with readline() and sleep() to emulate “tail -f”.
Questions
-
Is there an easier way of “tail -f” a whole bunch of textfiles directly onto a RabbitMQ stream? Something inbuilt, or an extension we could leverage on? Any other tips/advice here?
-
If we do write a Python wrapper to capture all the logfiles and publish them – I’d ideally like a single Python script to concurrently handle all the logfiles, rather than manually spinning up a separate instance for each logfile. How should we tackle this? Are there considerations in terms of performance, CPU usage, throughput, concurrency etc.?
- We need to subscribe to the queues, and then possibly dump the events back to disk and reconstruct the original logfiles. Any tips/advice on this? And we’d also like a single Python script we could startup to handle reconstructing all of the logfiles – rather than 50 separate instances of the same script – is that easily achievable?
Cheers,
Victor
PS: We did have a look at Facebook’s Scribe, as well as Flume, and both seem a little heavyweight for our needs.
You seem to be describing centralized syslog with rabbitmq as the transport.
If you could live with syslog, take a look at syslog-ng. Otherwise, you might
save some time by using parts of logstash ( http://logstash.net/ ).