Question

I have a very big text file (few GB) that has the following format:

1 2
3 4
3 5
3 6
3 7
3 8
3 9

File is already sorted and double lines were removed. There are repeated pairs like ‘2 1’, ‘4 3’ reverse order that I want to remove. Does anybody have any solution to do it in a very resource limited environments, in BASH, AWK, perl or any similar languages? I can not load the whole file and loop between the values.

Answer 1

Possible solution:

• Scan the file
• For any pair where the second value is less than the first, swap the two numbers
• Sort the pairs again by first then second number
• Remove duplicates

I’m still thinking about more efficient solution in terms of disk sweeps, but this is a basic naive approach