Home/ Questions/Q 966369
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-16T02:08:47+00:00

I have several csv’s that look like this: I have several large text files

  • 0

I have several csv’s that look like this:

I have several large text files (csv’s) that on some lines have redundant entries. That is, due to the way they were merged a certain field will often have the same value twice or three times. It’s not always in the same order though.

BWTL, NEWSLETTER, NEWSLETTER
BWTL, NEWSLETTER, R2R, NEWSLETTER
MPWJ, OOTA HOST, OOTA HOST, OOTA HOST
OOTA HOST, ITOS, OOTA HOST

Etc. The entries that are next to each other are easy enough to clean up with sed

sed -i “” ‘s/NEWSLETTER, NEWSLETTER/NEWSLETTER/g’ *.csv

Is there a similar quick way to fix up the other duplicates?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    You could do something like

    sed -i "" 's/^\(.*NEWSLETTER.*\), NEWSLETTER/\1/g' eNewsletter.csv_new.csv

    It works by capturing everything up to the second NEWSLETTER ^ means beginning of line
    \( and \) delimit the capture, and .* denotes anything at all. It then replaces the matched string with just the part captured.

    • 0