Question

I’m trying to parse big *.gz file using Perl in Windows.

In Solaris I’m able to use following construction:

my $cmd = "zcat $dir/$file|";
open FILE, $cmd or die "$cmd:$!";

while (<FILE>) {
.
.
.
}

and it works fine.
In Windows I’m tring to use IO::Zlib module, e.g.

my $fh = IO::Zlib->new("$file", "rb");
while (my $line = $fh->getline()) {
.
.
}

but I’m running out of memory. (I have 4GB RAM on my system). Is there any other method to parse big *.gz file?

Answer 1

Why not just install Gzip for Windows (which includes zcat)? Aside from the memory issue you’re having, I’ve found that piping from gzip is faster than using IO::Zlib. (There’s a couple reasons for that. $fh->getline is a method call, and Perl’s method calls aren’t the fastest. Also, running gzip externally takes advantage of multitasking, which can be noticeable now that multi-core machines are common.)

For some reason, GnuWin provides zcat as a shell script, which doesn’t really work under Windows. But you can use gzip -cd instead of zcat (that should work on Unix platforms, too).