Question

I have a large array of hashes, about 0.5Gb, stored in memory, and I need to remove some elements from it, about 10% spread around the whole array.

What is likelier to work best, doing a grep, or identifying the elements that need deleting, and splice-ing them out?

Thanks,

Simone

Answer 1

splice may go to O(n^2) under conditions you describe (as it moves array content around), and grep/slice would allocate O(n) extra memory (probably, much less than 500GB, but still…).

There is a linear solution with no extra memory though, but looks more like C than like Perl:

sub inplace_grep {
    my ($code, $array) = @_; 
    # move elements backwards
    for (my ($to, $from)=(0,0); $from < @$array; $from++) {
        $code->($array->[$from]) or next;
        $array->[$to++] = $array->[$from];
    };
    # remove tail 
    splice @$array, $to; 
};

Update: on grep memory usage — you can do a quick test for extra memory allocation by using large amounts of data and looking for brk syscall. On my system (linux, perl 5.10) it does.

strace -e trace=brk perl -MTime::HiRes -wle \
'print "start ".time; my @array = 1..10**7; print "alloc ".time;
@array = grep { $_ %2 } @array; print "grep ".time'