Question

When running the below file with no output redirection, the output is as expected.

Output

./get_urls.pl
www.site1.com
www.site2.com
www.siten.com

When redirecting STDOUT to a file, nothing is recorded in the file.

./get_urls.pl > out
cat out

–

#!/usr/bin/perl

use LWP::Simple;
use strict;
use warnings;

my $i = 1;

while (my $contents = get("http://www.validpage.com?page=$i"))
{
        #print STDERR $contents."\n".$url."\n";

        #print STDERR $i."\n";

        my @matches = ($contents =~ /_full'>(.*)?</g);

        for my $match (@matches)
        {
                $match =~ s/\s//g;
                print $match."\n";
        }

        $i++;
}

print STDERR "$i total matches.\n";

I suspect this behavior is a side effect of using LWP::Simple because output redirects as expected when the get() function call is omitted.

Answer 1

Add

$| = 1;

to the beginning of your code. It’s autoflush issue when output is not sent to the terminal.

When STDOUT is a terminal, perl autoflushes the output, but if STDOUT is, say, pipe like that

perl print_urls.pl |cat

it is no more a TTY but a bare file descriptor and autoflush is set to 0 to optimize file operations (you don’t want you logfile flushed after writing each line, right?)

Even with autoflush disabled you will see the output… Sooner or later. Usually buffer is 4096 bytes so when your program output reaches 4097th char you will see first 4096 bytes at once.

Also, if you don’t what to enable autoflush globally you can also try

use IO::Handle;
STDOUT->autoflush(1);

More reading on that is man perlvar and perldoc IO::Handle