Question

I have a text file with some rows in the following form

*,[anything, even blanks],[dog|log|frog],[dog|log|frog],[0|1],[0|1],[0|1]

I would like to remove duplicate rows that have the same value for * (case insensitive), ie anything left of ,[anything, even blanks],[dog|log|frog],[dog|log|frog],[0|1],[0|1],[0|1]

For example here’s a sample text file

test,bar,log,dog,0,0,0
one
foo,bar,log,dog,0,0,0
/^test$/,bar,log,dog,0,0,0
one
FOO,,frog,frog,1,1,1

The resulting text file should have the duplicate foo removed (order does not matter to me so long as the duplicates are removed, leaving 1 unique)

test,bar,log,dog,0,0,0
one
/^test$/,bar,log,dog,0,0,0
one
FOO,,frog,frog,1,1,1

What’s the simplest bash command I could do to achieve this?

Answer 1

awk -F, '!seen[tolower($1)]++' file