I’m working on a pretty big enterprise application using Perl, has tens of modules, etc, which mainly used to crawl some stuff over the web.
One of the subroutines I wrote, is doing an image retrieval & analysis. Usually it takes couple of seconds to accomplish, for each parameter I sent to it. So I’m sending it to a different process (forking…). The issue is, after some time the system becomes very unstable, memory filled up.
Questions:
- Is it because each process created, creates a copy of the parent data in a separate memory location? if so, does it mean each child has a copy of ALL the modules? (and there are tens…)
- What is the best approach to free this memory / manage these processes?
Forking uses copy-on-write, so forked processes shouldn’t grab too much memory unless they are especially long-lived.
It sounds like you’ve got a memory leak in your system. Do you have any mutually-recursive dependencies or circular data structures? If so, you may want to look into using Scalar::Util::weaken to tweak reference-counts on your data structures.