Question

I have two files. the first (file1.txt) has about 15,000 lines and the second file (file2.txt) has about 180,000 lines. I am trying to find all of the matching lines from file1.txt and remove them from file2.txt. What I am currently trying is this:

grep -v -f file1.txt file2.txt > out.txt

This does work, but takes an extremely long time. I am trying to find a way to speed this up, but am having trouble figuring it out. Does anyone have a suggestions?
Thank for your help

Answer 1

if the definition of “match” here is “identical”:

try this:

awk 'NR==FNR{a[$0];next;}!($0 in a)' file1 file2 >out.txt