Question

I have a big log file in which the records are sorted by time. Each line has a time. I need to find all the records between time T1 and time T2 (T1 <= T2). I can scan the whole file line by line and find the start line with T1, copy that into a buffer and then scan the next line until I hit the end time T2. This will work but not very efficient.

I wonder if I can use binary search to locate the lines with time T1 and T2. But I am not sure how to determine the following:

1. The middle line of the file
2. How to determine the offset we should pass to lseek()?

Is that possible to use binary search on a file?

Answer 1

Let us assume, that your lines are all reasonable near to the average length (meaning there is no line that will take up half of the log or so), which will make binary search feasible.

Next I will also assume you will have following functions:

//find the first start of a new log line after (or including) position start
//return the last position of the file if no start could be found
streampos findNextLineStart(ifstream &file, streampos start);
//extract the data as a timestamp from a line
int extractDate(ifstream &file, streampos lineStart);

With these functions we can implement the following:

//find the position of the first line whose date is bigger than the given
streampos lower_bound(ifstream &file, int date)
{
  file.seekg(0, ios::end);
  streampos begin = 0, 
            end = file.tellg();
  while(begin < end)
  {
     streampos cur = (begin + end) / 2;
     streampos start = findNextLineStart(file, cur);
     //was a line start found?
     if(start < end)
     {
        int lineDate = extractDate(file, start);
        if(lineDate < date)
          begin = start;
        else
          end = start;
     }
     else
       //narrow the bound as no line was found
       end = cur;
  }
  return begin;
}

I do not guarantee for this to work (on all corner cases) but it sketches the overall implementation. One would use another function upper_bound and with those you could get start and end of the lines that are within your bounds.