Question

I have lines in a log file, appended chronologically. For example, it could be data from the last 30 days, starting 30 days ago, then 29 days ago, then 28 days ago, etc.

I want to read the file in normal chronological order, but starting from a certain point (e.g., starting 7 days ago, read 7 days ago data, then 6 days ago data, then 5 days ago data, etc.)

One method is just reading the file normally, however for speed reasons I will need to:
– seek from the end of the file backward, exponentially, to find the right point to start at
– then, once I found the right point to start at, read lines one by one, in forward order

I’m having trouble getting this to work. I started by modifying the answer here:
Most efficient way to search the last x lines of a file in python

Can someone help, or provide guidance on a better way to do this?

Answer 1

If speed is a concern, that probably means you are doing it many times, or have to do it on-the-fly. Thus, you could build an index file showing the position you have to seek to for each day, something like:

Day 1: 0
Day 2: 1048576
Day 3: 2097152
Day 4: 6291456
....

This would allow fast lookup of any day once the index is built.

The algorithm for updating this index would be to start at the position of the last known day, read forward, and each time you reach a new day add it to the index.