Question

We are developing a database tool and we’d like to write a log file in a format which is extendable and easy to be imported into a database table. We all feel that filtering this info by using SQL is a good idea, since the log will be a long file and “search” may not be good enough. Could you give me some suggestions? Any experiences will be useful too! Thanks in advance.

Answer 1

The first thing I would say is that your file format ought to be human readable. My reasons are given here: Why should I use a human readable file format.

Beyond that, it is impossible to answer with such a vague question. However, here are some of the issues you should consider:

1. How big does this log file grow? How does this compare to the space you have? If space is going to be an issue, then a more parsimonious format is better – eg Protocol Buffers.
2. How is the log file going to be looked at? If it is using specific tools, the format matters less than if you are going to be using a text editor or excel
3. What sort of data are you storing? If it is just ASCII text then CSV works well.
4. Is type information important in your data? Do you need to compare numbers and dates as numbers and dates rather than just strings? If so then some sort of typed system (eg XML or JSON) might be better
5. Is the data going to be transferred to other people? In which case something with good language tools for reading and writing might be important
6. How quickly does the data need to written? If speed is an issue (which it might be for realtime log files) then a format optimised for this might be important.
7. How quickly does the data need to be read?
8. Will all the data need to be in memory, or can it be scanned in a serialized way?

When you can answer all these questions, you’ll probably know the answer yourself. If not, make your question more specific with these questions answered and it will be easier for someone to help you.

Personally I’ve always been grateful when log data has been written as CSV. It is flexible enough to expand (add extra columns, change the length of a field), is quick to read and write in to a database spreadsheet, and hundreds of other tools, and is codeable in seconds. However, it does have a number of disadvantages – it is verbose, easy to get escapes wrong, untyped, and easy to break if you rearrange columns.