I have a long running rake task that is gobbling up all my system memory over time? What is the quickest way to track down my issue and get to the bottom of it?
Im using rails 2.3.5, ruby 1.8.7, ubuntu on slicehost and mysql 5.
I have rails app that works fine. I have a nightly job that runs all night and does tons of work (some external calls to twitter, google etc, and lots of db calls using active record, over time that job grows in memory size to nearly 4 gig. I need to figure out why the rake task is not releasing memeory.
I started looking into bleak_house, but it seems complex to setup and hasnt been updated in over a year. I cant get it to work locally so im reluctant to try in production.
thanks
Joel
Throwing out two ideas. First, if you’re looping as part of this job, make sure you’re not holding onto references to objects you don’t need, as this will prevent them from being collected. If you’re done, remove them from your array, or whatever. Also, put a periodic GC.start into your loop as a way to see if it’s simply not getting around to GC-ing.
Second idea is that ruby does not GC symbols, so if your API clients are storing values as symbols you can end up with a huge and growing set of symbols that will never be re-used. Symbols are tiny, but tiny things can still add up.
And of course, don’t load more objects than you need to. use #find_each to load AR objects in batches if you have to iterate over lots of them.