I’m building a large import script that uses functionality from a separate code base that I suspect of having a memory leak. It calls the code base as many as 10000 times for the same operations and while the first is relatively quick (2 sec) the script is requiring a long time to run (over 100 hours and counting) and by the end the same task is up to 60 sec or more (and still climbing). What is the best way to work around this while the leaks are found and fixed?
Some solutions that have been brainstormed would be:
Create a process that runs a part of the script then end it, reclaiming the resources it used.
Use a shell script to launch the program multiple times completing a sub-set of the tasks each time and have the updated data output to file to be used by the next iteration
edit: Changed the way the question was phrased to make it clear that the import and the code base are separate programs
I solved this issue by minimizing the scope that contains references to the other codebase. Basically every time I initialize an object or call a function from the other codebase I went through hoops to make sure it existed for the minimal time possible. Often setting references again to NULL in order to make sure all references were removed.
This ended up working excellently, reduced the time from over 150 hours and counting to under 30.