Is there a way to delete duplicate lines in a file in Unix?
I can do it with sort -u and uniq commands, but I want to use sed or awk.
Is that possible?
Is there a way to delete duplicate lines in a file in Unix? I
Share
Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.
Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.
Lost your password? Please enter your email address. You will receive a link and will create a new password via email.
Please briefly explain why you feel this question should be reported.
Please briefly explain why you feel this answer should be reported.
Please briefly explain why you feel this user should be reported.
Is there a way to delete duplicate lines in a file in Unix?
I can do it with sort -u and uniq commands, but I want to use sed or awk.
Is that possible?
seenis an associative array that AWK will pass every line of the file to. If a line isn’t in the array thenseen[$0]will evaluate to false. The!is the logical NOT operator and will invert the false to true. AWK will print the lines where the expression evaluates to true.The
++incrementsseenso thatseen[$0] == 1after the first time a line is found and thenseen[$0] == 2, and so on.AWK evaluates everything but
0and""(empty string) to true. If a duplicate line is placed inseenthen!seen[$0]will evaluate to false and the line will not be written to the output.