Question

I am currently working on a script which processes csv files, and one of the things it does is remove and keep note of duplicate lines in the files. My current method to do this is to run uniq once using uniq -d once to display all duplicates, then run uniq again without any options to actually remove the duplicates.
Having said that, I was wondering if it would be possible to perform this same function in one action instead of having to run uniq twice. I’ve found a bunch of different examples of using awk to remove duplicates out there, but as far as I know I have not been able to find any that both displayed the duplicates and removed them at the same time.
If anyone could offer advice or help for this I would really appreciate it though, thanks!

Answer 1

Here’s something to get you started:

awk 'seen[$0]++{print|"cat>&2";next}1' file > tmp && mv tmp file

The above will print any duplicated lines to stderr at the same time as removing them from your input file. If you need more, tell us more….