Question

I have multiple “tables” in a file, such as:

col1, col2, col3, col4
1, 2, 3, 4
5, 6, 7, 8

col2, col3, col5
10, 11, 12
13, 14, 15

And I would like to collapse these 2 tables to:

col1, col2, col3, col4, col5
1   , 2   , 3   , 4   , 
5   , 6   , 7   , 8   , 
    , 10  , 11  ,     , 12
    , 13  , 14  ,     , 15

(Note: extra whitespace left just to make things easier to understand)

This would seem to require at least 2 passes, one to collect the full list of columns, and another one to create the output table. Is it possible to do this with awk? If not, what other tool would you recommend?

Answer 1

The code assumes that the tables are separated by empty lines:

awk -F', *' 'END {
  for (i = 0; ++i <= c;)
    printf "%s", (cols[i] (i < c ? OFS : RS))
  for (i = 0; ++i <= n;)
    for (j = 0; ++j <= c;)
      printf "%s", (vals[i, cols[j]] (j < c ? OFS : RS))    
  }
!NF { 
  fnr = NR + 1; next 
  }
NR == 1 || NR == fnr  {
 for (i = 0; ++i <= NF;) {
   _[$i]++ || cols[++c] = $i
   idx[i] = $i
   }
  next 
  }
{  
  ++n; for (i = 0; ++i <= NF;)
         vals[n, idx[i]] = $i
   }' OFS=', ' tables

If you have the tables in separate files:

awk -F', *' 'END {
  for (i = 0; ++i <= c;)
    printf "%s", (cols[i] (i < c ? OFS : RS))
  for (i = 0; ++i <= n;)
    for (j = 0; ++j <= c;)
      printf "%s", (vals[i, cols[j]] (j < c ? OFS : RS))    
  }
FNR == 1 {
 for (i = 0; ++i <= NF;) {
   _[$i]++ || cols[++c] = $i
   idx[i] = $i
   }
  next 
  }
{  
  ++n; for (i = 0; ++i <= NF;)
         vals[n, idx[i]] = $i
   }' OFS=', ' file1 file2 [.. filen]