I’ve got a web application that, for performance reasons, throws any data sent into a logfile.
I’ve got two concerns with this approach:
- How do I best rotate logs, in order to not lose data?
- For each user session multiple requests are logged. Each request has a unique id so there is an easy way for me to tie the requests to the session. The problem is, however, that if I rotate the logs I risk ending up with one request in one log and another request in another log.
How do I arrange my parsing in a way that allows me to parse all requests from a given session? I am willing to define a session timelimit, for example that the requests must, at maximum be 30 minutes apart.
If I had a hourly log rotation at 00 minutes:
What if the user made one request at 13:59 and one at 14:01 – The user would end up having requests in two different logs.
Answer to part 1: If you’re on *nix, use syslog/logger. Check the logger(1) and syslog.conf(5) man pages.
Answer to part 2: You’re not forced to look at just one log file at a time.
less ${SERVICE}*will normally open all the relevant log files together: when you get to the bottom of a page, :n will move you to the next file and :p back.Alternatively, use a log analyser program. Steve Kemp’s post on promptly finding needles in syslog haystacks covers, together with its comments, a lot of ground.