Question

i have a file like this:

term1 term2
term3 term4
term2 term1
term5 term3
..... .....

what i need to do is to remove duplicates in any order they appear, such as:

term1 term2

and

term2 term1

is a duplicate to me.
It is a really long file, so I’m not sure what can be faster.
Does anyone has an idea on how to do this? awk perhaps?

Answer 1

Ordering each word in the line and sorting is easy with perl.

./scriptbelow.pl < datafile.txt | uniq

#!/usr/bin/perl

foreach(sort map { reorder($_) } <>) {
    print;
}

sub reorder {
    return join(' ', sort { $a cmp $b } split(/\s+/, $_)) . "\n";
}