Question

I’ve been writing output from perl scripts to files for some time using code as below:

open( OUTPUT, ">:utf8", $output_file ) or die "Can't write new file: $!";

print OUTPUT "First line I want printed\n";
print OUTPUT "Another line I want printing\n";

close(OUTPUT);

This works, and is faster than my initial approach which used “say” instead of print (Thank you NYTProf for enlightening my to that!)

However, my current script is looping over hundreds of thousands of lines and is taking many hours to run using this method and NYTProf is pointing the finger at my thousands of ‘print’ commands. So, the question is… Is there a faster way of doing this?

Other Info that’s possibly relevant…
Perl Version: 5.14.2 (On Ubuntu)

Background of the script in question…
A number of ‘|’ delimited flat files are being read into hashes, each file has some sort of primary key matching entries from one to another. I’m manipulating this data and them combining them into one file for import into another system.

The output file is around 3 Million lines, and the program starts to noticeably slow down after writing around 30,000 lines to said file. (A little reading around seemed to point towards running out of write buffer in other languages but I couldn’t find anything about this with regard to perl?)

EDIT: I’ve now tried adding the line below, just after the open() statement, to disable print buffering, but the program still slows around the 30,000th line.

OUTPUT->autoflush(1);

Answer 1

I think you need to redesign the algorithm your program uses. File output speed isn’t influenced by the amount of data that has been output, and it is far more likely that your program is reading and processing data but not releasing it.

• Check the amount of memory used by your process to see if it increases inexorably

• Beware of for (<$filehandle>) loops, which read whole files into memory at once

• As I said in my comment, disable the relevant print statements to see how performance changes