Question

I have many large (~30 MB a piece) tab-delimited text files with variable-width lines. I want to extract the 2nd field from the nth (here, n=4) and next-to-last line (the last line is empty). I can get them separately using awk:

awk 'NR==4{print $2}' filename.dat

and (I don’t comprehend this entirely but)

awk '{y=x "\n" $2};END{print y}' filename.dat

but is there a way to get them together in one call? My broader intention is to wrap it in a Python script to harvest these values from a large number of files (many thousands) in separate directories and I want to reduce the number of system calls. Thanks a bunch –

Edit: I know I can read over the whole file with Python to extract those values, but thought awk might be more appropriate for the task (having to do with one of the two values located near the end of the large file).

Answer 1

Here’s how to implement this in Python without reading the whole file

To get the nth line, you have no choice but to read the file up to the nth line as the lines are variable width.

To get the second to last line, guess how long the line might be (be generous) and seek to that many bytes before the end of the file.

read() from the point you have seeked to. Count the number of newline characters – You need at least two. If there are less than 2 newlines double your guess and try again

split the data you read at newlines – the line you seek will be the second to last item in the split