Question

I have the following problem that I want to solve in awk. I have one large text table, comma separated, consisting of 100k rows and 5k cols. The first row is a header and the first column is a record id. I then have a second text file that contains a subset of the headers in the first file. I want to extract all the columns of the first file whose header is contained in the list given in the second file. Here an example of the inputs and the desired output:

DATA.TXT

   ID, head1, head2, head3, head4  
    1, 25.5, 1364.0, 22.5, 13.2  
    2, 10.1, 215.56, 1.15, 22.2  

LIST.TXT

head1  
head4  

Desired output:

ID, head1, head4  
1, 25.5, 13.2  
2, 10.1, 22.2

Anybody can give me some advice on how to solve this problem in awk or however through unix scripting? Thanks in advance for any help!

Answer 1

I have an idea, but since I’m not experienced in shell programming (and don’t know awk) this looks like reinventing some wheels in a ridiculous way:

$ cat DATA.TXT 
ID, head1, head2, head3, head4
1, 25.5, 1364.0, 22.5, 13.2
2, 10.1, 215.56, 1.15, 22.2

$ cat LIST.TXT 
head1
head4

$ cols=($(sed '1!d;s/, /\n/g' DATA.TXT | grep -nf LIST.TXT | sed 's/:.*$//'))

$ cut -d ',' -f 1$(printf ",%s" "${cols[@]}") DATA.TXT 
ID, head1, head4
1, 25.5, 13.2
2, 10.1, 22.2

P.S. I used some very basic ideas about bash arrays from this and this answers.