Question

I have 2 tabular files. One file contains a mapping of 50 key values only called lookup_file.txt.
The other file has the actual tabular data with 30 columns and millions of rows. data.txt
I would like to replace the id column of the second file with the values from the lookup_file.txt..

How can I do this? I would prefer using awk in bash script..
Also, Is there a hashmap data-structure i can use in bash for storing the 50 key/values rather than another file?

Answer 1

Assuming your files have comma-separated fields and the “id column” is field 3:

awk '
BEGIN{ FS=OFS="," }
NR==FNR { map[$1] = $2; next }
{ $3 = map[$3]; print }
' lookup_file.txt data.txt

If any of those assumptions are wrong, clue us in if the fix isn’t obvious…

EDIT: and if you want to avoid the (IMHO negligible) NR==FNR test performance impact, this would be one of those every rare cases when use of getline is appropriate:

awk '
BEGIN{
   FS=OFS=","
   while ( (getline line < "lookup_file.txt") > 0 ) {
      split(line,f)
      map[f[1]] = f[2]
   }
}
{ $3 = map[$3]; print }
' data.txt