Question

Here is the simple perl script fetching data from SQL.
Read data and write on a file OUTFILE, and print the data on screen for every 10000th line.

One thing I am curious is that the printing the data on screen terminates very quickly(in 30 seconds), however, data fetching and writing on a file ends very slowly(30 minutes later).

The amount of data is not large. The output files size is less than 100Mbyte.

while ( my ($a,$b) = $curSqlEid->fetchrow_array() ) 
{
    printf OUTFILE ("%s,%d\n", $a,$b);
    $counter ++;
    if($counter % 10000 == 0){
        printf ("%s,%d\n", $a,$b);
    }
}  
$curSqlEid->finish();
$dbh->disconnect();
close(OUTFILE);

Answer 1

You are suffering from buffering.

Handles other than STDERR are buffered by default, and most handles use a block buffering. That means Perl will wait until there is 8KB* of data to write before sending anything to the system.

STDOUT is special. When is attached to a terminal (and only then), it uses a different kind of buffering: line buffering. When using line buffering, the data is flushed every time a newline is encountered in the data to write.

You can see this by running

$ perl -e'print "abc"; print "def"; sleep 5; print "\n"; sleep 5;'
[ 5 seconds pass ]
abcdef
[ 5 seconds pass ]

$ perl -e'print "abc"; print "def"; sleep 5; print "\n"; sleep 5;' | cat
[ 10 seconds pass ]
abcdef

The solution is to turn off buffering.

use IO::Handle qw( );  # Not needed on Perl 5.14 or later
OUTFILE->autoflush(1);

* — 8KB is the default. It can be configured when Perl is compiled. It used to be a non-configurable 4KB until 5.14.