Question

The following python script prints the first occurrence of a line in a file.

#!/usr/bin/env python
import sys

x = set() 

for line in sys.stdin:
    if line not in x:
        print line,
        x.add(line)

uniq only works for adjacent lines. The file this will be used on is very large so sort | uniq is not ideal. Is there a standard tool that does this?

Answer 1

Have you tried sort -u? At least that’s what the uniq man page suggests.