Question

I have a tool that generates tests and predicts the output. The idea is that if I have a failure I can compare the prediction to the actual output and see where they diverged. The problem is the actual output contains some lines twice, which confuses diff. I want to remove the duplicates, so that I can compare them easily. Basically, something like sort -u but without the sorting.

Is there any unix command line tool that can do this?

Answer 1

uniq(1)

SYNOPSIS

uniq [OPTION]… [INPUT [OUTPUT]]

DESCRIPTION

Discard all but one of successive identical lines from INPUT (or standard input), writing to OUTPUT (or standard output).

Or, if you want to remove non-adjacent duplicate lines as well, this fragment of perl will do it:

while(<>) {
    print $_ if (!$seen{$_});
    $seen{$_}=1;
}