Question

I need your help. Say I have a code inside a for statement similar to as shown below.

the value of $logCount is massive. Like a million. There are certain hashes that are made in my loop that grows over period of time, and it runs my system out memory!. What I am supposed to do is, run through the loop and then write the results to a file. But since I am running out of memory this never happens. Hence I want to break my loop into steps of 1000.

Can you help? Is there a smarter way to do this? I dont know how to append to the bottom of the file, if I break my loop.

for (my $i=0; $i < $logCount; $i++){    
    # Crap code
    # Herp Derp
    generatePowerRecord($sectorMapper->{'sectors'}, \$powerresult, ,\%Dimensions, \@powers, \@Attn, $sectors, $count-$totalcount);
    generatePhaseRecord($sectorMapper->{'sectors'}, \$phaseresult, ,\%Dimensions);
    generateDelayRecord($sectorMapper->{'sectors'}, \$delayresult, ,\%Dimensions, \@delay_history, \$sectors, $count-$totalcount);
};

$fh->print($dataresult);
$fh->print($powerresult);
$fh->print($phaseresult);
$fh->print($delayresult);
$fh->print("\n}");  

Answer 1

The simplest solution is to throw extra prints into your loop and call them every 1000th iteration, something like this:

for (my $i=0; $i < $logCount; $i++){    
    # Crap code
    # Herp Derp
    generatePowerRecord($sectorMapper->{'sectors'}, \$powerresult, ,\%Dimensions, \@powers, \@Attn, $sectors, $count-$totalcount);
    generatePhaseRecord($sectorMapper->{'sectors'}, \$phaseresult, ,\%Dimensions);
    generateDelayRecord($sectorMapper->{'sectors'}, \$delayresult, ,\%Dimensions, \@delay_history, \$sectors, $count-$totalcount);

    # call this every 1000th iteration
    if($i > 0 and $i % 1000 == 0) {
       $fh->print($dataresult);
       $fh->print($powerresult);
       $fh->print($phaseresult);
       $fh->print($delayresult);

       # cleanup hashes
       undef $dataresult;
       undef $powerresult;
       undef $phaseresult;
       undef $delayresult;
    }
};

The print call will append data to your file until it remains opened.