Question

I have a .csv file with approx 10 columns that is logging data. I want to use something like this:

How to get the first line of a file in a bash script?

Where it is grabbing the first line of each file and then processing the logs. However, once the line is processed, I want to mark it as processed (can be as simple as adding a new column on the end with a *** or something). So basically, I want to grab the first line not processed, process it, and move to the next unprocessed line, etc.

I need to do this using sed, awk, grep, and/or other standards. The bash script will sit and run in the background on an infinite while loop. Essentially, I am trying to read and process this log file in real-time, but need the log for history.

Edit:
Also, I need this to mark the lines that have been read in the file. That way if the server stops, I can know right where to pick up processing. So tail will work if I can figure out a way to do that.

Thanks!

Answer 1

Rather than sitting in a infinite loop you could do this:

tail -n +1 -f your_log_file | some_processing_pipeline

This will start reading your logfile at line 1, then continuously wait for new lines to appear and pass them to some_processing_pipeline.