I am using IPython 0.10.2 and Python 2.7 right now. I start one ipcontroller and 20 ipengines on my cluster. The code structure is very simple. I just use MultiEngineClient.execute() methods and MultiEngineClient dictionary interface (e.g., mec[‘a’] = b) . My current application needs to run nearly two days. However, after 10 hours running, I find that ipcontroller has used 3.1 GB memory. I do use MultiEngineClient.clear_properties(). But it never releases any memory. Does anyone know how to release the memory of ipcontroller ?
Share
Unfortunately, it may not be possible to free memory effectively in the Controller in IPython 0.10, but the
MultiEngineClient.clear_pending_results()method may help.The controller in 0.11-0.12 can use a database (sqlite or mongodb) to store results, and has been seen to run for a long time, with gigabytes of throughput without unreasonable long-term growth.