Question

Here I want to have your help on trimming a file, by remove the rows which with the same value in the columns except the first two columns.

the file I have (tab-delimited, with millions of rows, and tens of columns)

Jack Mike Jones Dan Was
1 2 7 3 4
2 3 9 4 8
T T C T T
T M T T T
W A S I S

the file I want (remove the rows which have the same values in cells except the first two)

Jack Mike Jones Dan Was
1 2 7 3 4
2 3 9 4 8
T T C T T
W A S I S

Could you give me any hints on my problem? Thanks a lot.

And I have experienced several excellent scripts of awk, shell and perl, in a related question. Thanks a lot for the helpers.

Answer 1

awk '{
    val=$3
    for (i=4; i<=NF; i++)
        if (val != $i) {
            print
            break
        }
}'