Question

If I have an application that writes a log file to the filesystem, and I later want to be able to dump those logs into a RDBMS to be able to query easier on specific items (e.g. I want to query all logs related to transaction ID X, not just queries like everything with a log level of INFO or since some date).

What is a robust way of formatting the log messages to be able to easily put it in a RDBMS to allow for future changes in what I am actually logging (adding more things)?

Answer 1

Code the attributes of the log entry as name-value pairs. You will need to make a choice about the delimiter for the value part; that is, it needs to be something that won’t appear in the value itself.

Some attributes of the log entry, if you haven’t thought about that part of the problem:

• timestamp
• severity (fatal, error, info, debug), probably as a string
• location in the source code from which the entry was made, as a file name and line number and/or class name and method name
• transaction
• free text: whatever the code needs to record in the entry

Also consider:

• stack trace: kind of messy, since it tends to be an unspecified iteration
• exception information provided by your runtime (C#? Java?)
• user ID
• thread info

It’s that free text attribute that will compete with your delimiter choice.

A toy example log entry, using double brackets as a delimiter:

timemstamp=[[24-Apr-2012 15:48:28.749]] severity=[[error]] location=[[Foo.java (45)]] transaction=[[48241]] free=[[Cannot open file 'zqx3.txt']]

Your question is not tagged with an implementation language. In Java land, log4j is a popular framework for logging. I believe there are suitable ports or work-alikes for .NET.

See also A simple log file format.