Question

I’ve got a DOS program kicked off by a cmd script that dumps a lot of data I’d like to track in a log file. I can easily pipe it to a file – but I need a rolling logfile.

Is there an easy way to pipe the output to a program that will generate a daily rolling logfile? (ie a new file is created for each day)

Answer 1

If you don’t control the source for the program you’re running, one possibility is to have another script, run as a scheduled task, which shuts down the program, moves the log file, and restarts the program. Schedule this for somewhere around midnight.

If you can’t shut down the program periodically, another possibility is to develop a filter program which will take standard input and send it to a log file based on today’s date. That program can detect date changes and close/reopen the file when it crosses the midnight boundary.

Then pipe the output of your original program through this filter. This is true piping by the way. What you’re currently doing is not technically piping but redirection (piping goes to a process, redirection goes to a file).

Pseudocode would be something like:

lastDate = ""
currFile = null
while not end of file on standard input:
    get line from standard input
    currDate = getDate()
    if currDate not equal to lastDate:
        if currFile not null:
            close currFile
        currFile = open("prefix_"+currDate+".log")
    write line to currFile
if currFile not null:
    close currFile
exit

---

As a proof of concept, here’s a script (qqtest.sh) that runs, generating the date every thirteen seconds, ten times:

#!/usr/bin/bash
for i in 0 1 2 3 4 5 6 7 8 9 ; do
    echo $i $(date)
    sleep 13
done

And here’s a C filter program (qq.c) that does what I describe in my answer above:

#include <stdio.h>
#include <string.h>
#include <stdlib.h>
#include <time.h>

static char *getDate (void) {
    static char tbuff[sizeof("yyyymmddhhmm")]; // "yyyymmdd" for date.
    time_t now = time (0);
    struct tm *tm = localtime (&now);
    strftime (tbuff, 100, "%Y%m%d%H%M", tm);   // "%Y%m%d" for date.
    return tbuff;
}

int main (void) {
    int ch, lastCh = '\n';
    FILE *fOut = NULL;
    char *currDate, *lastDate = strdup ("");
    char fspec[1000];

    // Just process characters until finished.

    while ((ch = fgetc (stdin)) != EOF) {
        // Only switch logs if first character of a line.

        if (lastCh == '\n') {
            // Has date changed?

            currDate = getDate();
            if (strcmp (currDate, lastDate) != 0) {
                // Yes, close old file if there was one.

                if (fOut != NULL)
                    fclose (fOut);

                // Then store new date and open new file.

                free (lastDate);
                lastDate = strdup (currDate);
                sprintf (fspec, "qqfile_%s.log", lastDate);
                fOut = fopen (fspec, "w");
            }
        }

        // Output character to current file then save.

        fputc (ch, fOut);
        lastCh = ch;
    }

    // Clean up memory and file handles, then exit.

    free (lastDate);
    if (fOut != NULL)
        fclose (fOut);

    return 0;
}

When you execute:

./qqtest.sh | ./qq

it creates the following files.

$ cat qqfile_201005211146.log
0 Fri May 21 11:46:40 WAST 2010
1 Fri May 21 11:46:53 WAST 2010

$ cat qqfile_201005211147.log
2 Fri May 21 11:47:06 WAST 2010
3 Fri May 21 11:47:19 WAST 2010
4 Fri May 21 11:47:33 WAST 2010
5 Fri May 21 11:47:46 WAST 2010
6 Fri May 21 11:47:59 WAST 2010

$ cat qqfile_201005211148.log
7 Fri May 21 11:48:12 WAST 2010
8 Fri May 21 11:48:25 WAST 2010
9 Fri May 21 11:48:38 WAST 2010

Note that this uses a minute boundary to switch log files. It’s a simple matter to change the getDate function to use the day boundary instead (see the comments).