Question

I am calling a webservice that’s returning a comma separated dataset with varying columns and multiple text-qualified rows (the first row denotes the column names) . I need to insert each row into a database while concatenating the rows that are varied.

The data is returned like so

"Email Address","First Name","Last Name", "State","Training","Suppression","Events","MEMBER_RATING","OPTIN_TIME","CLEAN_CAMPAIGN_ID"

"[e-mail]","Scott","Staph","NY","Campaigns and activism","Social Media","Fundraiser",1,"2012-03-08 17:17:42","Training"

There can be up to 60 columns between State and Member_Rating, and the data in those fields are to get concatenated and inserted into one database column. The first four fields and the last three fields in the list will always be the same. I’m unsure the best way to tackle this.

Answer 1

I am not sure if this solution fits your needs. I hope so. It’s a perl script that joins with - surrounded with spaces all fields but first four and last three. It uses a non standard module, Text::CSV_XS that must be installed using CPAN or similar tool.

Content of infile:

"Email Address","First Name","Last Name","State","Training","Suppression","Events","MEMBER_RATING","OPTIN_TIME","CLEAN_CAMPAIGN_ID"
"[e-mail]","Scott","Staph","NY","Campaigns and activism","Social Media","Fundraiser",1,"2012-03-08 17:17:42","Training"

Content of script.pl:

use warnings;
use strict;
use Text::CSV_XS;

my $csv = Text::CSV_XS->new({ 
        allow_whitespace => 1,
});

open my $fh, q[<], $ARGV[0] or die qq[Open: $!\n];

while ( my $row = $csv->getline( $fh ) ) {
        my $concat = join q[ - ], (@$row)[4 .. @$row-4];
        splice @$row, 4, scalar @$row - (3 + 4), $concat;
        $csv->print( \*STDOUT, $row );
        print qq[\n];
}

Run it like:

perl script.pl infile

With following output:

"Email Address","First Name","Last Name",State,"Training - Suppression - Events",MEMBER_RATING,OPTIN_TIME,CLEAN_CAMPAIGN_ID
[e-mail],Scott,Staph,NY,"Campaigns and activism - Social Media - Fundraiser",1,"2012-03-08 17:17:42",Training