Question

I’m designing a daemon that will continuously read lines from a single text file and process those lines. What is a good general purpose way to keep track of the last line processed, independent of the file name, in the event of lines being written to the text file while the daemon isn’t running?

Every so often, the file is archived and a new blank file is created in its place. The daemon will be stopped for the archival to occur.

My first idea, which seems overcomplicated, is to compute and store a hash and line number of the last successfully processed record. Then, when the daemon is started again, run to that line number and calculate the hash. If the hash matches, continue on processing the next record. If the hash doesn’t match, start over on the file at the beginning, since that will say this is a new file.

I have a feeling there is a good general purpose technique used by log file analyzers or something in a text book that I haven’t had exposure to.

Answer 1

Assuming you have permission, enough disk space and assuming you kill the daemon safely…
Just write the last line processed to a file (upon shutdown of the daemon).

You could wrap each instance of the daemon inside a context manger if you want

from contextlib import contextmanager

http://docs.python.org/library/contextlib.html

class a_daemon():
    def __init__(self,last_line):
        print "initilizing.."
        self.last_line=last_line

    def run_me(self):
        print "running.."
        #while true, process lines, set last_line to current line being processesed.
        self.last_line='blah'

from contextlib import contextmanager

@contextmanager
def run_new_daemon():
    print "getting last line"
    last_line=open("last_line.txt").read() #you should get a "file does not exist" error the first time running this unless you created the file already

    my_daemon=a_daemon(last_line)
    yield my_daemon

    print "shutting down, writing last line to file."
    with open("last_line.txt",'w') as last_line_file:
        last_line_file.write(my_daemon.last_line)

with run_new_daemon() as my_daemon:
    my_daemon.run_me()