Question

I have a text file that is 310MB in size (uncompressed). When using PerlIO::gzip to open the file and uncompress it into memory, this file easily fills 2GB of RAM before perl runs out of memory.

The file is opened as below:

open FOO, "<:gzip", "file.gz" or die $!;
my @lines = <FOO>;

Obviously, this is a super convenient way to open gzipped files easily in perl, but it takes up a ridiculous amount of space! My next step is to uncompress the file to the HD, read the lines of the file to @lines, operate on @lines, and compress it back. Does anyone have any idea why over 7 times as much memory is consumed when opening a zipped file? Does anyone have an alternate idea as to how I can uncompress this gzipped file into memory without it taking a ridiculous amount of memory?

Answer 1

When you do:

my @lines = <FOO>;

you are creating an array with as many elements as there are lines in file. At 100 characters per line, that’s about 3.4 million array entries. There is overhead associated with each array entry which means the memory footprint will be much larger than just the uncompressed size of the file.

You can avoid slurping and process the file line-by-line. Here is an example:

C:\Temp> dir file
2010/10/04  09:18 PM       328,000,000 file

C:\Temp> dir file.gz
2010/10/04  09:19 PM         1,112,975 file.gz

And, indeed,

#!/usr/bin/perl

use strict; use warnings;
use autodie;
use PerlIO::gzip;

open my $foo, '<:gzip', 'file.gz';

while ( my $line = <$foo> ) {
    print ".";
}

has no problems.

To get an idea of the memory overhead, note:

#!/usr/bin/perl

use strict; use warnings;
use Devel::Size qw( total_size );

my $x = 'x' x 100;
my @x = ('x' x 100);

printf "Scalar: %d\n", total_size( \$x );
printf "Array:  %d\n", total_size( \@x );

Output:

Scalar: 136
Array:  256