Question

I am using the paste command in a bash loop to add new columns to a CSV-file. I would like to reuse the CSV-file. Currently I am using a temporary file to accomplish this:

while [ $i -le $max ]
    do
        # create text from grib2
        wgrib2 -d 1.$(($i+1)) -no_header myGribFile.grb2 -text tmptxt.txt

        #paste to temporary file
        paste -d, existingfile.csv tmptxt.txt > tmpcsv.csv  

        #overwrite old csv with new csv
        mv tmpcsv.csv existingfile.csv

        ((i++))
    done

After adding some columns the copy is getting slow, because the file is becoming bigger and bigger (every tmptxt.txt has about 2 MB, adding to approx 100 MB).

A tmptxt.txt is a plain txt-file with one column and one value per row:

1
2
3
.
.

The existingfile.csv would then be

1,1,x
2,2,y
3,3,z
.,.,.
.,.,.

Is there any way to use the paste command to add a column to an existing file? Or is there any other way?

Thanks

Answer 1

Would it be feasible to split the operation in 2 ? One step for generating all the intermediate files; and another for generating all the final output file. The idea is to avoid rereading and rewriting over and over the final file.

The changes to the script would be something like this:

while [ $i -le $max ]
do
    n=$(printf "%05d" $i)    # to preserve lexical order if $max > 9
    # create text from grib2
    wgrib2 -d 1.$(($i+1)) -no_header myGribFile.grb2 -text tmptxt$n.txt
    ((i++))
done

#make final file
paste -d, existingfile.csv tmptxt[0-9]*.txt > tmpcsv.csv  

#overwrite old csv with new csv
mv tmpcsv.csv existingfile.csv