We are currently considering moving from logging to files, to logging to a NoSQL database. Another team in our company is starting to use HBase but that looked quite complex for what we would like to do. I have been looking at MongoDB but i would like some suggestions.
Currently we have hundreds of servers in farms for different applications. Whenever we hear of a problem from one of the applications there is a long process the retrieve the logs from app ops and then a long process to sift through them all to find the problem. We are looking at just moving all of our logging into a central place and building a web UI around it so we can access and query the logs more easily.
Which NoSQL database would be a good fit for storing and querying applications log?
I’ve seen a lot of companies are using MongoDB to store application logs. Its schema-freeness is really flexible for application logs, at which schema tends to change time-to-time. Also, its Capped Collection feature is really useful because it automatically purges old data to keep the data fit into the memory.
People aggregates the logs by normal Grouping or MapReduce, but it’s not that fast. Especially MongoDB’s MapReduce only works within a single thread and its JavaScript execution overhead is huge. New aggregation framework could solve this problem.
When you use MongoDB for logging, the concern is the lock contention by high write throughputs. Although MongoDB’s insert is fire-and-forget style by default, calling a lot of insert() causes a heavy write lock contention. This could affect the application performance, and prevent the readers to aggregate / filter the stored logs.
One solution might be using the log collector framework such as Fluentd, Logstash, or Flume. These daemons are supposed to be launched at every application nodes, and takes the logs from app processes.
They buffer the logs and asynchronously writes out the data to other systems like MongoDB / PostgreSQL / etc. The write is done by batches, so it’s a lot more efficient than writing directly from apps. This link describes how to put the logs into Fluentd from PHP program.
Here’s some tutorials about MongoDB + Fluentd.
MongoDB’s problem is it starts slowing down when the data volume exceeds the memory size. At that point, you can switch to other solutions like Apache Hadoop or Cassandra. If you have a distributed logging layer mentioned above, you can switch to another solution as you grow. This tutorial describes how to store logs to HDFS by using Fluentd.