Home/ Questions/Q 6250605
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-24T13:27:23+00:00

I am tasks with comparing 2 large unsorted .csv files based on column 1

  • 0

I am tasks with comparing 2 large unsorted .csv files based on column 1 and 3.
Each file contains about 200k records. For the output, I need to know which records based on column 1 and 3 exist in the first file but not the second file. The files are quoted comma separated value files. Column 3 needs to ignore case when comparing.

Example File1: 

"id", "name", "email", "country"
"1233",  "jake", "jake@mailinator.com", "USA"
"2345", "alison", "Alison@mailinator.com", "Canada"
"3456", "jacob", "jacob@mailinator.com", "USA"
"5678", "natalia", "natalia@mailinator.com", "USA"

File 2

"id", "name", "email", "country"
"2345", "alison", "alison@mailinator.com", "Canada"
"3456", "jacob", "jacob@mailinator.com", "USA"
"5690", "lina", "lina@mailinator.com", "Canada"

desired Output file

"5678", "natalia", "natalia@mailinator.com", "USA"

Code examples would be very appreciated.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Try:

    join -v 1 -i -t, -1 1 -2 1 -o 1.2 1.3 1.4 1.5  <(awk -F, '{print $1":"$3","$0}' f1.txt | sort) <(awk -F, '{print $1":"$3","$0}' f2.txt | sort)

    How it works:

    1) I first create a composite key column, by joining column 1 and column3:

    awk -F, '{print $1":"$3","$0}' f1.txt
awk -F, '{print $1":"$3","$0}' f2.txt

    2) I sort both outputs:

    awk -F, '{print $1":"$3","$0}' f1.txt | sort 
awk -F, '{print $1":"$3","$0}' f2.txt | sort

    3) I then use the join command to join on the first column (my composite key) and output the unpairable lines coming from file 1.

    Output:

    "1233",  "jake", "jake@mailinator.com", "USA"
"5678", "natalia", "natalia@mailinator.com", "USA"
    • 0