Question

I am building a program to poll a network device every 48 hours and record the output user Perl and the Expect module. At the moment, I am saving the output text to a text file and then filtering the text file to get the required information. Originally, this was fine since efficiency wasn’t that big of a problem when dealing with file that are only a hundred KB in size and keeping most of the text anyway, but I’ve found a new command I want to execute remotely on the network device. This command takes an hour and a half to run, creates a text file around 5MB in size and I would probably only be keeping 10KB of information. Using basic file I/O (read a text file, write to a text file) just seems like a bad idea and I keep thinking that there has to be a better way.

What my research has turned up so far:
I should probably be using some sort of Pipe. Something like:

open($filehandle, "myCommand");

There are other solutions out there, but they seem to imply that I save the ENTIRE file to one variable, when I would like to only save and modify parts of it before writing to a file.
TIEHANDLE works well for overwiting "print", but I don't think that's necessary or will work here.

What I've tried so far:
Redirecting STDOUT using a pipe. Result: Kept on sending the output to a file and couldn't edit the text before it was put in the file.

I have run out of things to Google. If a complete solution is too daunting of a task, a hint as to where the next steps might be would also be greatly appreciated.

Also, if the advantages in terms of processing time are actually minimal (I'm unclear on how long it takes to process a 5mb test file and will investigate), please let me know.

Answer 1

Perl can crunch through a 5MB file in no time, I would not worry about trying to redirect the output to a pipe when you can just open the file and parse through it once your job is done running.

    open(INPUT, "/path/to/file.log") or die "Can't open file: $!";

    while (<INPUT>) {
    chomp;
      if (/ERROR/) {
        print "Found an error!";
      }
    }
    close(INPUT);
    exit;

When you’re in the while() loop, you can use regex, split, or whatever you want to find what you’re looking for in the file. The current line is stored in $_. You will get one iteration for each line of your input file.

Hope this helps…