Question

I made the following Perl script to handle some file manipulation at work, but it’s running far too slowly at the minute to be put in production.

I don’t know Perl very well (not one of my languages), so can someone help me identify and replace parts of this script that would be slow given it’s processing ~40 million lines?

Data being piped in is in the format:

col1|^|col2|^|col3|!|
col1|^|col2|^|col3|!|
... 40 million of these.

The date_cols array is calculated before this part of the script and basically holds the index of columns containing dates in the pre-converted format.

Here’s the part of the script that will be executed for every input row. I’ve cleaned it up a little and added comments, but let me know if anything else is needed:

## Read from STDIN until no more lines are arailable.
while (<STDIN>)
{       
    ## Split by field delimiter
    my @fields = split('\|\^\|', $_, -1);   

    ## Remove the terminating delimiter from the final field so it doesn't
    ## interfere with date processing.
    $fields[-1] = (split('\|!\|', $fields[-1], -1))[0];

    ## Cycle through all column numbres in date_cols and convert date
    ##  to yyyymmdd
    foreach $col (@date_cols)
    {
        if ($fields[$col] ne "")
        {
            $fields[$col] = formatTime($fields[$col]);
        }
    }

    print(join('This is an unprintable ASCII control code', @fields), "\n");
}           

## Format the input time to yyyymmdd from 'Dec 26 2012 12:00AM' like format.
sub formatTime($)
{
    my $col = shift;        

    if (substr($col, 4, 1) eq " ") {
        substr($col, 4, 1) = "0";
    }       
    return substr($col, 7, 4).$months{substr($col, 0, 3)}.substr($col, 4, 2);
}

Answer 1

If written purely for efficiency, I’d write your code like this:

sub run_loop {
  local $/ = "|!|\n"; # set the record input terminator
                      # to the record seperator of our problem space
  while (<STDIN>) {       
    # remove the seperator
    chomp;

    # Split by field delimiter
    my @fields = split m/\|\^\|/, $_, -1;

    # Cycle through all column numbres in date_cols and convert date
    #  to yyyymmdd
    foreach $col (@date_cols) {
      if ($fields[$col] ne "") {
        # $fields[$col] = formatTime($fields[$col]);
        my $temp = $fields[$col];
        if (substr($temp, 4, 1) eq " ") {
          substr($temp, 4, 1) = "0";
        }       
        $fields[$col] = substr($temp, 7, 4).$months{substr($temp, 0, 3)}.substr($temp, 4, 2);
      }
    }
    print join("\022", @fields) . "\n";
  }
}

The optimizations are:

• Using chomp to remove the |!|\n string at the end
• Inlining the formatTime sub.

Subroutine calls are extremely expensive in Perl. If subs have to be used very efficiently, prototype checking can be disabled with the &subroutine(@args) syntax. If @args are ommited, the current arguments @_ are visible to the called sub. This can lead to bugs or additional performance. Use wisely. The goto &subroutine; syntax can be used as well, but this meddles with return (basically a tail call). Do not use.

Further optimizations could include removing the hash lookup %months, as hashing is expensive.