I’m searching for tool/database/solution that can help me with aggregating real-time logs and can query them also in real-time.
Basic requirement is ability to deliver results as soon as possible, keeping in mind, that there might be many of events to query (possibly billions), but logs would have many ‘columns’ and each query would set some conditions on those columns, so final result will be some kind of aggregation, or only small subset of rows would be returned.
Right now I was looking at HDFS+HBase which seems like a good solution. Are there any alternatives? Can you recommend anything?
You can check Flume: https://github.com/cloudera/flume/wiki .