Question

I wrote (with lots of help from others) an awk command to total up a column in a CSV file. Unfortunately, I learned after some Googling that awk isn’t great at handling CSV files due to the fact that the separator is not always the same (i.e. commas should be ignored when surround by quotes).

It seems that perhaps a Perl script could do better. Would it be possible to have a one-line Perl script (or something nearly as succinct) that achieves the same thing as this awk command that totals up the 5th column of a CSV file?

cat file.csv | awk -F "\"*,\"*" '{s+=$5} END {printf("%01.2f\n", s)}'

I’m not married to Perl in particular but I was hoping to avoid writing a full-blown PHP script. By this time I could have easily written a PHP script, but now that I’ve come this far, I want to see if I can follow it through.

Answer 1

You need to use a decent CSV parser to deal with all the complexities of CSV format. Text::CSV_XS (or Text::CSV if that’s not avialable) is one of the preferred ones.

perl -e '{use Text::CSV_XS; my $csv=Text::CSV_XS->new(); open my $fh, "<", "file.csv" or die "file.csv: $!"; my $sum = 0; while (my $row = $csv->getline ($fh)) {$sum += $row->[4]}; close $fh; print "$sum\n";}'

Here’s the actual Perl code, for better readability

use Text::CSV_XS; # use the parser library
my $csv = Text::CSV_XS->new(); # Create parser object
open my $fh, "<", "file.csv" or die "file.csv: $!"; # Open the file. 
my $sum = 0; 
while (my $row = $csv->getline ($fh)) { # $row is array of field values now
    $sum += $row->[4];
}
close $fh; 
print "$sum\n";

---

The above could be shortened by using slightly lesser quality but denser Perl:

cat file.csv | perl -MText::CSV_XS -nae '$csv=Text::CSV_XS->new(); 
               $csv->parse($_); @f=$csv->fields(); $s+=$f[4]} { print "$s\n"'