Question

I need to parse several large size XML files (one is ~8GB, others are ~4MB each) and merge them. Since both SAX and Tie::File are not suitable due to memory and time issues, I decided to try Twig.

Suppose each XML file is composed of several elements as follows:

<class name=math>
     <student>luke1</student>
     ... (a very very long list of student)
   <student>luke8000000</student>
</class>
<class name=english>
   <student>mary1</student>
     ...
   <student>mary1000000</student>
</class>

As you see, even if I use TwigRoots => {"class[\@name='english']" => \&counter} I still need to wait a long time for Twig to start to parse class=english because it needs to go over each line of class=math first (correct me if it does not need to go over each line).

Is there any way to let Twig start the parsing from a line number, rather than the beginning of a file? I can get the line number of <class name = english> using grep, which is much faster.

Thanks in advance.

Answer 1

Perhaps this example will give you some ideas for an alternative strategy. In particular, you might be able to combine the idea in index_file with Zoul’s suggestion about seeking to a location before passing off the file handle to XML::Twig.

use strict;
use warnings;

# Index the XML file, storing start and end positions
# for each class in the document. You pay this cost only once.
sub index_file {
    local @ARGV = (shift);
    my (%index, $prev);
    while (<>){
        if ( /^<class name=(\w+)>/ ) {
            my $start = tell() - length();
            $index{$1} = { start => $start, end => undef };

            $index{$prev}{end} = $start - 1 if defined $prev;
            $prev = $1;
        }        
        $index{$prev}{end} = tell if eof;
    }
    return \%index;
}

# Use the index to retrieve the XML for a particular class.
# This allows you to jump quickly to any section of interest.
# It assumes that the sections of the XML document are small enough
# to be held in memory.
sub get_section {
    my ($file_name, $class_name, $index) = @_;
    my $ind = $index->{$class_name};

    open(my $fh, '<', $file_name) or die $!;    
    seek $fh, $ind->{start}, 0;
    read( $fh, my $xml_section, $ind->{end} - $ind->{start} );

    return $xml_section;
}

# Example usage.
sub main {
    my ($file_name) = @_;
    my $index = index_file($file_name);
    for my $cn (keys %$index){
        # Process only sections of interest.
        next unless $cn eq 'math' or $cn eq 'english';
        my $xml = get_section($file_name, $cn, $index);

        # Pass off to XML::Twig or whatever.
        print $xml;
    }
}

main(@ARGV);