Question

We’ve got a mature body of code that loads data from files into a database.
There are several file formats; they are all fixed-width fields.

Part of the code uses the Perl unpack() function to read fields from the input data into package variables.
Business logic is then able to refer to these fields in a ‘human-readable’ way.

The file reading code is generated from a format description once, prior to reading a file.

In sketch form, the generated code looks like this:

while ( <> ) {

    # Start of generated code.

    # Here we unpack 2 fields, real code does around 200.
    ( $FIELDS::transaction_date, $FIELDS::customer_id ) = unpack q{A8 A20};

    # Some fields have leading space removed
    # Generated code has one line like this per affected field.
    $FIELDS::customer_id =~ s/^\s+//;

    # End of generated code.

    # Then we apply business logic to the data ...
    if ( $FIELDS::transaction_date eq $today ) {
        push @fields, q{something or other};
    }

    # Write to standard format for bulk load to the database.
    print $fh join( '|', @fields ) . q{\n} or die;
}

Profiling the code reveals that around 35% of the time is spent in the unpack and leading-space strip.
The remaining time is spent in validating and transforming the data, and writing to the output file.

It appears that there is no single part of the business logic that takes more than 1-2% of the run time.

The question is – can we eke out a bit more speed from the unpacking and space stripping somehow? Preferably without having to refactor all the code that refers to the FIELDS package variables.

EDIT:

In case it makes a difference

$ perl -v
This is perl, v5.8.0 built for PA-RISC1.1

Answer 1

Yes. Extracting using substr is likely to be the fastest way to do it. That is:

$FIELDS::transaction_date = substr $_, 0, 8;
$FIELDS::customer_id      = substr $_, 8, 20;

is likely to be faster. Now, if I were handwriting this code, I would not give up unpack, but if you are generating the code, you might as well give it a shot and measure.

See also the answers to Is Perl’s unpack() ever faster than substr()?

As for stripping leading spaces, s/^\s+// is likely to be the fastest method.

Update: It is hard to say anything definite without being able to run benchmarks. However, how about:

my  $x  = substr $_, 0, 8;

for fields that do not need any trimming and

my ($y) = substr($_, 8, 20) =~ /\A\s+(.+?)\s+\z/;

that do need trimming?