Question

I have written a Perl program which will match certain words in a log file and store the results in a database. The problem is this program works fine with a small file but doesn’t work with file size ~2GB. Is it size or program need to be changed?

use POSIX qw(strftime);

# load module
use DBI;

open( FILE, "/root/temp.log" ) or die "Unable to open logfile:$!\n";
$count_start   = 0;
$count_interim = 0;
$count_stop    = 0;

while (<FILE>) {
  @test = <FILE>;
  foreach $line (@test) {

    if ( $line =~ m/server start/ ) {

      #print "yes\n";
      $count_start++;
    }
    elsif ( $line =~ m/server interim-update/ ) {
      $count_stop++;
    }
    elsif ( $line =~ m/server stop/ ) {
      $count_interim++;
    }

  }
  print "$count_start\n";
  print "$count_stop\n";
  print "$count_interim\n";
  $now_string = strftime "%b %e %H:%M:%S", localtime;
  print $now_string;

  # connect
  my $dbh = DBI->connect( "DBI:Pg:dbname=postgres;host=localhost",
    "postgres", "postgres", { 'RaiseError' => 1 } );

  # execute INSERT query
  my $rows = $dbh->do(
"insert into radcount (acc,bcc,dcc) Values  ('$count_start','$count_stop','$count_interim')"
  );

  print "$rows row(s) affected\n";

  # clean up
  $dbh->disconnect();

}

close(LOG);

Answer 1

I have a few comments about your program.

• Always use strict and use warnings at the start of your program, and declare variables using my at their point of first use

• Always use lexical filehandles and the three-parameter form of open, and always check the status of an open call

• You are opening the file using filehandle FILE, but closing LOG

• Your while statement reads the first line of the file and throws it away

• @test = <FILE> attempts to read all of the rest of the file into the array. This is what is causing your problem

• You should connect to the database once and use the same database handle for the rest of the code

• You should use prepare your statement with placeholders and pass the actual values with execute

• You are incrementing $count_stop for an interim-update record and $count_interim for a stop record

• The core module Time::Piece provides a strftime method without the bloat of POSIX

Here is a modification of your program to show these ideas. I have not set up a log file and database to test it but it looks fine to me and does compile.

use strict;
use warnings;

use Time::Piece;
use DBI;

open my $log, '<', '/root/temp.log' or die "Unable to open log file: $!";

my ($count_start, $count_interim, $count_stop) = (0, 0, 0);

while (<$log>) {

  if ( /server start/ ) {
    $count_start++;
  }
  elsif ( /server interim-update/ ) {
    $count_interim++;
  }
  elsif ( /server stop/ ) {
    $count_stop++;
  }
}

print <<END;
Start:   $count_start
Interim: $count_interim
Stop:    $count_stop
END

print localtime->strftime("%b %e %H:%M:%S"), "\n";

my $dbh = DBI->connect(
    "DBI:Pg:dbname=postgres;host=localhost", "postgres", "postgres",
    { 'RaiseError' => 1 } );

my $insert = $dbh->prepare('INSERT INTO radcount (acc, bcc, dcc) VALUES (?, ?, ?)');
my $rows = $insert->execute($count_start, $count_stop, $count_interim);

printf "%d %s affected\n", $rows, $rows == 1 ? 'row' : 'rows';