Question

My apologies if this is too basic. I’m really not looking for someone to do the work but rather point me in the right direction. I have a log files going back several years that I want to pull information out to determine how long something is taking to look for patterns on when performance is slow. I am able to read each line but cannot read the previous line to get its time.

The Log files are as follows:

~
Other Stuff
~
12/21/11 18:58:15 Inserting data into ST_ITEMS
ST_ITEMS Row: 10000 inserted at 12/21/11 19:40:06
ST_ITEMS Row: 20000 inserted at 12/21/11 20:05:58
ST_ITEMS Row: 30000 inserted at 12/21/11 20:37:03
ST_ITEMS Row: 40000 inserted at 12/21/11 20:59:25
ST_ITEMS Row: 50000 inserted at 12/21/11 21:17:43
ST_ITEMS Row: 60000 inserted at 12/21/11 21:54:47
12/21/11 21:59:24 Finished inserting data into  Staging Tables

~
Other Stuff
~

12/21/11 22:04:43 Inserting data into ST_ITEMS
ST_ITEMS Row: 10000 inserted at 12/21/11 22:38:53
ST_ITEMS Row: 20000 inserted at 12/21/11 23:06:33
ST_ITEMS Row: 30000 inserted at 12/21/11 23:33:03
ST_ITEMS Row: 40000 inserted at 12/22/11 00:05:38
ST_ITEMS Row: 50000 inserted at 12/22/11 00:45:59
ST_ITEMS Row: 60000 inserted at 12/22/11 01:12:42
ST_ITEMS Row: 70000 inserted at 12/22/11 01:40:02
ST_ITEMS Row: 80000 inserted at 12/22/11 02:14:23
ST_ITEMS Row: 90000 inserted at 12/22/11 03:04:15
ST_ITEMS Row: 100000 inserted at 12/22/11 03:47:13
ST_ITEMS Row: 110000 inserted at 12/22/11 04:36:21
ST_ITEMS Row: 120000 inserted at 12/22/11 05:44:47
ST_ITEMS Row: 130000 inserted at 12/22/11 06:28:24
ST_ITEMS Row: 140000 inserted at 12/22/11 07:10:55
ST_ITEMS Row: 150000 inserted at 12/22/11 07:35:16
12/22/11 07:40:28 Finished inserting data into  Staging Tables

~
Other Stuff
~

Essentially, I want to calculate how long each 10000 rows take by subtracting the date/time of a line from the one above it. I’m looking at both Perl and Bash as options but it seems that Perl offers more possibilities.

PERL
#!/usr/bin/perl

use strict;
use warnings;

use Date::Parse;
use Date::Format;

my $start = "2007-11-17 12:50:22";
my $stop  = "2007-11-17 12:53:22";
my $diff  = str2time($stop) - str2time($start);

#printf "diff between %s and %s is %d seconds\n", $start, $stop, $diff;

open(LOG,"info_refresh_tvl.log.122111_185800") or die "Unable to open logfile:$!\n";
while(my $line = <LOG>){


        if ($line=~/inserted\b/)

        {
        #Pseudocode  
            #Parse time from Pervious Line
            #Parse time from Current Line
            #Calculate Difference of Time
                    #my $diff  = str2time($stop) - str2time($start);
            #printf "diff between %s and %s is %d seconds\n", $start, $stop,     $diff; ')

            printf $line ;


        }

}
close(LOG);

BASH

grep 'ST_ITEMS Row:' logfile122111.log | while read line
   do
        event=$(echo "$line" | awk '{print $6 " " $7}')

       case $event in
          "10000")
                ;;
          *)
                past=$(echo "$line" | awk '{print $6 " " $7}')
                current=$(echo "$line" | awk '{print $6 " " $7}'
                echo $past
                echo $current)
                ;;
       esac



echo $event


   done

Answer 1

Just save each line when you continue after the comparison. Overwrite it with the current line after you’re done.

In pseudocode:

$CurrentLine = $line;
#Parse time from $CurrentLine
#Parse time from $LastLine
#Calculate difference of time
$LastLine = $line;