Question

This sample script:

#!/usr/bin/perl -w

while (1)
{
  sleep(1);
}

takes about 264 kB

grep -A1 heap  /proc/9216/smaps 
0817b000-081bd000 rw-p 0817b000 00:00 0          [heap]
Size:               264 kB

but when I only add my module:

#!/usr/bin/perl -w

use my_module;

while (1)
{
  sleep(1);
}

it takes 18092 kB !

grep -A1 heap  /proc/9219maps 
0817b000-09326000 rw-p 0817b000 00:00 0          [heap]
Size:             18092 kB

Note: The ‘my_module’ has a lot of ‘use module;’ inside it.

How can I find what takes so much memory ?

How can I reduce it ? (using ‘use module (function)’ ?)

Thanks for your help.

Answer 1

Insert BEGIN {} blocks to narrow down the culprit as in

#! /usr/bin/perl

sub grep_heap { print @_, "\n"; system "grep -A1 heap /proc/$$/smaps" }

BEGIN { grep_heap "<null>" }
use warnings;
BEGIN { grep_heap "+warnings" }
use strict;
BEGIN { grep_heap "+strict" }

use Data::Dumper;
BEGIN { grep_heap "+Data::Dumper" }
use POSIX;
BEGIN { grep_heap "+POSIX" }

print "Hi\n";

On my Linux host, I see

$ ./prog.pl
<null>
0889b000-088de000 rw-p 0889b000 00:00 0                                  [heap]
Size:               268 kB
+warnings
0889b000-08920000 rw-p 0889b000 00:00 0                                  [heap]
Size:               532 kB
+strict
0889b000-08920000 rw-p 0889b000 00:00 0                                  [heap]
Size:               532 kB
+Data::Dumper
0889b000-089a4000 rw-p 0889b000 00:00 0                                  [heap]
Size:              1060 kB
+POSIX
0889b000-08ace000 rw-p 0889b000 00:00 0                                  [heap]
Size:              2252 kB
Hi

As for what to do about it, you could implement replacement modules with scaled down functionality or ask yourself whether you really need a particular module at all. However, in general, perl’s design prefers to throw memory at problems, and these days it’s common for machines to have multiple gibibytes of main memory.

Is this resource issue causing performance problems?

Below is a program that reads through the list of modules in perlmodlib.pod and for each module forks a child to require and import it and check its own heap.

#! /usr/bin/perl

sub heap {
  my($heap) = @_;
  unless ($heap =~ /^([0-9a-f]+)-([0-9a-f]+)/m) {
    warn "$0: unexpected heap:\n$heap";
    return -1;
  }
  hex($2) - hex($1);
}

sub size {
  my($bytes) = @_;

  my @units = (
    [ MiB   => "%.1f", 1_024 * 1_024 ],
    [ KiB   => "%.1f", 1_024 ],
  );

  for (@units) {
    my($unit,$fmt,$n) = @$_;
    return sprintf "$fmt %s" => $bytes/$n, $unit
      if $bytes >= $n;
  }

  return "$bytes byte" . ($bytes == 1 ? "" : "s");
}

my %incr;

my $perlmodlib = `perldoc -l perlmodlib`;
die "$0: perldoc failed" unless defined $perlmodlib;

my $base = heap `grep heap /proc/$$/smaps`;
warn "$0: base=" . size($base) . "\n";

chomp $perlmodlib;
open my $fh, "<", $perlmodlib   or die "$0: open $perlmodlib: $!";

while (<$fh>) {
  next unless /^=head2 Pragmatic Modules/ ..
              /^=head2 Extension Modules/;

  if (/^=item (\w+(::\w+)*)/) {
    my $mod = $1;
    (my $path = "$mod.pm") =~ s!::!/!g;

    my $pid = open my $fh, "-|";
    die "$0: fork: $!" unless defined $pid;

    if ($pid == 0) {
      open STDERR, ">", "/dev/null" or warn "$0: open: $!";
      exec "perl", "-e", <<EOProgram;
BEGIN {
  require \"$path\";
  eval { $mod->import };
  system qq(grep heap /proc/\$\$/smaps);
}
EOProgram
      die "$0: exec: $!";
    }
    else {
      local $/;
      my $heap = <$fh>;
      unless (defined $heap && length $heap) {
        warn "$0: use $mod failed";
        next;
      }
      $heap = heap $heap;
      $incr{$mod} = $heap > 0 ? $heap-$base : $heap;
    }
  }
}

foreach my $mod (sort { $incr{$b} <=> $incr{$a} } keys %incr) {
  print "$mod - ", size($incr{$mod}), "\n";
}

The top few offenders for perl-5.8.8:

CPAN::Nox - 9.7 MiB
CPAN - 9.6 MiB
ExtUtils::MM_VMS - 5.3 MiB
CPAN::FirstTime - 5.2 MiB
ExtUtils::Installed - 5.2 MiB
B::CC - 5.2 MiB
bigrat - 4.9 MiB
Math::BigRat - 4.8 MiB
ExtUtils::MM_NW5 - 4.7 MiB
ExtUtils::MM_OS2 - 4.6 MiB
ExtUtils::MM_Win32 - 4.6 MiB
ExtUtils::MM_Win95 - 4.6 MiB