Question

I’m iterating through a very large tab-delimited file (containing millions of lines) and pairing different lines of it based on the value of some field in that file, e.g.

mydict = defaultdict()
for line in myfile:
  # Group all lines that have the same field into a list
  mydict[line.field].append(line)

Since “mydict” gets very large, I’d like to make it into an iterator so I don’t have to hold it all in memory. How can I make it so instead of populating a dictionary, I will create an iterator that I can loop through and get all these lists of lines that have the same field value?

Thanks.

Answer 1

“millions of lines” is not very large unless the lines are long. If the lines are long you might save some memory by storing only positions in the file (.tell()/.seek()).

If the file is sorted by line.field; you could use itertools.groupby().

SQL’s GROUP BY might help for average-sized files (e.g., using sqlite as @wisty suggested).

For really large files you could use MapReduce.