Question

Whats the best way to summarize data from a file that has around 2 million records in Perl?

For eg: A file like this,

ABC|XYZ|DEF|EGH|100

ABC|XYZ|DEF|FGH|200

SDF|GHT|WWW|RTY|1000

SDF|GHT|WWW|TYU|2000

Needs to be summarized on the first 3 columns like this,

ABC|XYZ|DEF|300

SDF|GHT|WWW|3000

Chris

Answer 1

Assuming there are always five columns, the fifth of which is numeric, and you always want the first three columns to be the key…

use warnings;
use strict;

my %totals_hash;

while (<>)
{
  chomp;
  my @cols = split /\|/;

  my $key = join '|', @cols[0..2];

  $totals_hash{$key} += $cols[4];
}

foreach (sort keys %totals_hash)
{
  print $_, '|', $totals_hash{$_}, "\n";
}