Question

I am attempting to process a plain text file. It is basically an index of names and associated number fields formatted like so:

Nowosielski, Matthew, 484, 584, 777
Nowosielski, Matthew, 1151
Nunes, Paulino, 116
Nussbaum, Mike, 1221, 444,
Nussbaum, Mike, 156

Which I would like to process into this

Nowosielski, Matthew, 484, 584, 777, 1151
Nunes, Paulino, 116
Nussbaum, Mike, 156, 444, 1221

As you can see, the lines do not end consistently: some are likely to be whitespace, some newlines and some with commas. Effectively, I need to merge lines beginning with duplicated full-names, discarding the redundant name entry while merging and preserving the numerical order of the numerical fields.

My gut tells me to learn either some quick perl or awk, but my skill-set is, for both, empty. I looked into both, and after some searching and reading haven’t been able to find a clear or clean path to a solution.

My question thus is: what would be the best tool for the job that I might learn efficiently and just enough to complete this task? Also, given the suggested tool, are there any suggestions on how to approach the problem?

I can just edit this file by hand, of course, but that’s not very interesting and seems to be a very stupid, ham-fisted approach to the problem. I’m taking this task as an excuse to learn a bit about text processing as it feels like a problem for which there’s probably a good, existing tool.

Any pointers?

Answer 1

As Brian said, use a hash table. The following removes newlines, splits each record on commas, uses the “last name, first name” original form as a key to a hash, pushes the remaining values into an array and uses a reference to said array as the value to the above key.

Then it just iterates over the key/value pairs in the hash and formats accordingly.

Amended solution – sorting numbers, omitting middle names, and sorting output

#!/usr/bin/env perl
use strict;
use warnings;

my %merged;

while (my $record = <DATA>) {
    chomp $record;
    my ($lname, $fname, @stuff) = split /[, ]+/, $record;
    push @{ $merged{"$lname, $fname"} }, grep { m/^\d+$/; } @stuff;
}

foreach my $name (sort keys %merged) {
    print $name, ", ", join( ', ', sort { $a <=> $b } @{$merged{$name}}), "\n";
}

__DATA__
Nowosielski, Matthew, 484, 584, 777
Nowosielski, Matthew, 1151
Nunes, Paulino, 116
Nussbaum, Mike, 1221, 444,
Nussbaum, Mike, 156
Nowosielski, Matthew, Kimball, 485, 684, 277

Amended output

Nowosielski, Matthew, 277, 484, 485, 584, 684, 777, 1151
Nunes, Paulino, 116
Nussbaum, Mike, 156, 444, 1221

Original solution

#!/usr/bin/env perl
use strict;
use warnings;

my %merged;

while (my $record = <DATA>) {
    chomp $record;
    my ($lname, $fname, @stuff) = split /,/, $record;

    push @{ $merged{"$lname, $fname"} }, @stuff;
}

while (my ($name, $stuff) = each %merged) {
    print $name, join( ',', @$stuff), "\n"; 
}

__DATA__
Nowosielski, Matthew, 484, 584, 777
Nowosielski, Matthew, 1151
Nunes, Paulino, 116
Nussbaum, Mike, 1221, 444,
Nussbaum, Mike, 156