Question

I have a large (3GB), gzipped file containing two fields: NAME and STRING. I want to split this file into smaller files – if field one is john_smith, I want the string to be placed in john_smith.gz. NOTE: the string field can and does contain special characters.

I can do this easily in a for loop over the domains using BASH, but I’d much prefer the efficiency of reading the file in once using AWK.

I have tried using the system function within awk with escaped single quotes around the string

zcat large_file.gz | awk ‘{system(“echo -e ‘”‘”‘”$1″\t”$2″‘”‘”‘ | gzip >> “$1″.gz”);}’

and it works perfectly on most of the lines, however some of them are printed to STDERR and give an error that the shell cannot execute a command (the shell thinks that part of the string is a command). It looks like special characters might be breaking it.

Any thoughts on how to fix this, or any alternate implementations that would help?

Thanks!

-Sean

Answer 1

This little perl script does the job nicely

• keeping all destination files open for performance
• doing error elementary handling
• Edit now also pipes output through gzip on the fly

There is a bit of a kludge with $fh because apparently using the hash entry directly doesn’t work

#!/usr/bin/perl
use strict;
use warnings;

my $suffix = ".txt.gz";

my %pipes;
while (my ($id, $line) = split /\t/,(<>),2)
{
    exists $pipes{$id} 
        or open ($pipes{$id}, "|gzip -9 > '$id$suffix'") 
        or die "can't open/create $id$suffix, or cannot spawn gzip";

    my $fh = $pipes{$id};
    print $fh $line;
}

print STDERR "Created: " . join(', ', map { "$_$suffix" } keys %pipes) . "\n"

Oh, use it like

zcat input.gz | ./myscript.pl