Question

I’m writing a Perl script and I need to capture some lines from a garbage collection log and write them to a file.

The log is located on a remote host and I’m connecting using the Net::OpenSSH module.

I need to read the latest log file available.

In the shell I can locate the latest log with the following commands:

cd builds/5.7.1/5.7.1.126WRF_B/jboss-4.2.3/bin
ls -lat | grep '.log$' | tail -1

Which will return the latest log:

-rw-r--r--   1 load     other    2406173 Jul 11 11:53 18156.stdout.log

So in Perl I’d like to be able write something that locates and opens that log for reading.

When I have that log file, I want to print all lines that have a timestamp greater than a specified time. The specified timestamp is a $Runtime variable subtracted from the latest log message time.

Here are the last messages of the garbage collection log:

                                      ...

73868.629: [GC [PSYoungGen: 941984K->14720K(985216K)] 2118109K->1191269K(3065984K), 0.2593295 secs] [Times: user=0.62 sys=0.00, real=0.26 secs]
73873.053: [GC [PSYoungGen: 945582K->12162K(989248K)] 2122231K->1189934K(3070016K), 0.2329005 secs] [Times: user=0.60 sys=0.01, real=0.23 secs]

So if $Runtime had a value of 120 seconds, I would need to print all the lines from timestamp (73873.053 – 120) seconds up.

In the end my script would look something like this…

open GARB, ">", "./report/archive/test-$now/GC.txt" or die "Unable to create file: $!";

my $ssh2 = Net::OpenSSH->(
  $pathHost,
  user => $pathUser,
  password => $pathPassword
);
$ssh2->error and die "Couldn't establish SSH connection: ". $ssh2->error; 

# Something to find and open the log file.
print GARB #Something to return certain lines.
close GARB;

I realize this is somewhat similar to this question, but I can’t think of a way to tailor it to what I’m looking for. Any help is greatly appreciated!

Answer 1

Here’s an untested approach. I’ve not used Net::OpenSSH so there might be better ways to do it. I’m not even sure it works. What does work is the parsing part which I have tested.

use strict; use warnings;
use Net::OpenSSH;

my $Runtime = 120;
my $now = time;
open my $garb, '>', 
  "./report/archive/test-$now/GC.txt" or die "Unable to create file: $!";
my $ssh2 = Net::OpenSSH->(
$pathHost,
  user => $pathUser,
  password => $pathPassword
);
$ssh2->error and die "Couldn't establish SSH connection: ". $ssh2->error;   

# Something to find and open the log file.
my $fileCapture = $ssh2->capture(
  q~ls -lat builds/5.7.1/5.7.1.126WRF_B/jboss-4.2.3/bin |grep '.log$' |tail -1~
);
$fileCapture =~ m/\s(.+?)$/; # Look for the file name
my $filename = $1;           # And save it in $filename

# Find the time of the last log line 
my $latestTimeCapture = $ssh2->capture(
  "tail -n 1 builds/5.7.1/5.7.1.126WRF_B/jboss-4.2.3/bin/$filename");
$latestTimeCapture =~ m/^([\d\.]+):/;
my $logTime = $1 - $Runtime;

my ($in, $out, $pid) = $ssh2->open2(
  "builds/5.7.1/5.7.1.126WRF_B/jboss-4.2.3/bin/$filename");
while (<$in>) {
  # Something to return certain lines.
  if (m/^([\d\.]+):/ && $1 > $logTime) {
    print $garb $_; # Assume the \n is still in there
  }
}

waitpid($pid);

print $garb;
close $garb;

It uses your ls line to look up the file with the capture method. It then opens a pipe through the SSH tunnel to read that file. $in is a filehandle to that pipe which we can read.

Since we are going to process the file line by line, starting at the top, we need to first grab the last line to get the last timestamp. That is done with tail and, again, the capture method.

Once we have that, we read from the pipe line by line. This now is a simple regex (the same used above). Grab the timestamp and compare it to the time we have set earlier (minus the 120 seconds). If it is higher, print the line to the output filehandle.

The docs say we have to use waitpid on the $pid returned from $ssh2->open2 so it reaps the subprocess, so we do that before closing our output file.