Question

In our application, we log critical information to log text files for later debugging purpose. With splunk its easy to identify a problem if I already have some data points like order number or “object reference not found” type of error. But its challending for me to get an overall picture of a problem using splunk. To be able to identify an actual problem in software, I have to read through possibly multiple log files or an entire log file to see what application was doing before the problem happened. Reading entire log file in a human fashion helps me to identify how application behaved with other data points before the actual problem happened. In other words its hard for me to see the “real root cause” for the error with splunk. What has been your experience out there in the field of software development.

Answer 1

It’s very difficult to remove the human aspect. That being said, I’ve recently had to head the development side of a splunk rollout, and there are some fantastic tools to at least fulfill some of your needs. Using splunk’s built in alerts is the easiest way to do some of this. Unfortunately, there is a dearth of actual practical answers and examples for many splunk related things(i mean, seriously, do not use curl with an unsecured flag for every example of a webservice or rest api, please) in both splunkbase or elsewhere on the internet.

Either way, some of the most elegant solutions I’ve found for finding particular types of logs or log data has been heavy use of piping the “rex” command in my searches. It will specify Perl regexs for help in extracting the right information out of the right fields. Here’s the new-ish page on it from splunk’s website.

This of course assumes that you know what fields contain the data you’re looking for. Unfortunately, this can be an issue with windows logs if things are not set up correctly at the indexer.