Question

I need to slice several TB of log data, and would prefer the speed of the command line.
I’ll split the file up into chunks before processing, but need to remove some sections.

Here’s an example of the format:

uuJ oPz eeOO    109 66  8
uuJ oPz eeOO    48  0   221
uuJ oPz eeOO    9   674 3
kf iiiTti oP    88  909 19
mxmx lo uUui    2   9   771
mxmx lo uUui    577 765 27878456

The gaps between the first 3 alphanumeric strings are spaces. Everything after that is tabs. Lines are separated with \n.

I want to keep only the last line in each group.

If there’s only 1 line in a group, it should be kept.

Here’s the expected output:

uuJ oPz eeOO    9   674 3
kf iiiTti oP    88  909 19
mxmx lo uUui    577 765 27878456

How can I do this with sed, awk, xargs and friends, or should I just use something higher level like Python?

Answer 1

Try this:

awk 'BEGIN{FS="\t"}
    {if($1!=prevKey) {if (NR > 1) {print lastLine}; prevKey=$1} lastLine=$0}
    END{print lastLine}'

It saves the last line and prints it only when it notcies that the key has changed.