Question

What is the most efficient (fastest) way to simultaneously read in two large files and do some processing?

I have two files; a.txt and b.txt, each containing about a hundred thousand corresponding lines. My goal is to read in the two files and then do some processing on each line pair

def kernel:
    a_file=open('a.txt','r')
    b_file=open('b.txt', 'r')
    a_line = a_file.readline()
    b_line = b_file.readline()
    while a_line:
        process(a_spl,b_spl) #process requiring both corresponding file lines

I looked in to xreadlines and readlines but i’m wondering if i can do better. speed is of paramount importance for this task.

thank you.

Answer 1

The below code does not accumulate data from the input files in memory, unless the process function does that by itself.

from itertools import izip

def process(line1, line2):
  # process a line from each input

with open(file1, 'r') as f1:
  with open(file2, 'r') as f2:
    for a, b in izip(f1, f2):
      process(a, b)

If the process function is efficient, this code should run quickly enough for most purposes. The for loop will terminate when the end of one of the files is reached. If either file contains an extraordinarily long line (i.e. XML, JSON), or if the files are not text, this code may not work well.