Question

I have two huge files (file1 and file2). Both the files are organized into lines. I need to generate a third file file3 which has the lines that is there is file 1 but not in file 2. The lines are not ordered.

What is the easiest (smartest) way to get it in Windows?

Answer 1

The best strategy might depend on exactly how huge the files are. If the first file can fit into memory, then you can easily build a set of its lines, and removes lines from file2 from that set. This requires the amount of memory roughly proportional to the size file1.

with open('file1') as f1:
    lineset = set(f1)
with open('file2') as f2:
    lineset.difference_update(f2)
with open('file3', 'w') as out:
    for line in lineset:
        out.write(line)

Note that this solution will also eliminate duplicated lines from file1.