Question

I am planning on using delayed job to run some background analytics. In my initial test I saw tremendous amount of memory usage, so I basically created a very simple task that runs every 2 minutes just to observe how much memory is is being used.

The task is very simple and the analytics_eligbile? method always return false, given where the data is now, so basically none of the heavy hitting code is being called. I have around 200 Posts in my sample data in development. Post has_one analytics_facet.

Regardless of the internal logic/business here, the only thing this task is doing is calling the analytics_eligible? method 200 times every 2 minutes. In a matter of 4 hours my physical memory usage is at 110MB and Virtual memory at 200MB. Just for doing something this simple! I can’t even begin to imagine how much memory this will eat if its doing real analytics on 10,000 Posts with real production data!! Granted it may not run evevery 2 minutes, more like every 30, still I don’t think it will fly.

This is running ruby 1.9.7, rails 2.3.5 on Ubuntu 10.x 64 bit. My laptop has 4GB memory, dual core CPU.

Is rails really this bad or am I doing something wrong?

 Delayed::Worker.logger.info('RAM USAGE Job Start: ' + `pmap #{Process.pid} | tail -1`[10,40].strip)
Post.not_expired.each do |p|
    if p.analytics_eligible?
        #this method is never called
        Post.find_for_analytics_update(p.id).update_analytics
    end
end
Delayed::Worker.logger.info('RAM USAGE Job End: ' + `pmap #{Process.pid} | tail -1`[10,40].strip)

Delayed::Job.enqueue PeriodicAnalyticsJob.new(), 0, 2.minutes.from_now

Post Model

def analytics_eligible?
        vf = self.analytics_facet
        if self.total_ratings > 0 && vf.nil?
            return true
        elsif !vf.nil? && vf.last_update_tv > 0
            ratio = self.total_ratings / vf.last_update_tv
            if (ratio - 1) >= Constants::FACET_UPDATE_ELIGIBILITY_DELTA
                return true
            end
        end
        return false
    end

Answer 1

ActiveRecord is fairly memory-hungry – be very careful when doing selects, and be mindful that Ruby automatically returns the last statement in a block as the return value, potentially meaning that you’re passing back an array of records that get saved as a result somewhere and thus aren’t eligible for GC.

Additionally, when you call “Post.not_expired.each”, you’re loading all your not_expired posts into RAM. A better solution is find_in_batches, which specifically only loads X records into RAM at a time.

Fixing it could be something as simple as:

def do_analytics
  Post.not_expired.find_in_batches(:batch_size => 100) do |batch|
    batch.each do |post|
      if post.analytics_eligible?
        #this method is never called
        Post.find_for_analytics_update(post.id).update_analytics
      end
    end
  end
  GC.start
end

do_analytics

A few things are happening here. First, the whole thing is scoped in a function to prevent variable collisions from holding onto references from the block iterators. Next, find_in_batches retrieves batch_size objects from the DB at a time, and as long as you aren’t building references to them, become eligible for garbage collection after each iteration runs, which will keep total memory usage down. Finally, we call GC.start at the end of the method; this forces the GC to start a sweep (which you wouldn’t want to do in a realtime app, but since this is a background job, it’s okay if it takes an extra 300ms to run). It also has the very distinct benefit if returning nil, which means that the result of the method is nil, which means we can’t accidentally hang on to AR instances returned from the finder.

Using something like this should ensure that you don’t end up with leaked AR objects, and should vastly improve both performance and memory usage. You’ll want to make sure you aren’t leaking elsewhere in your app (class variables, globals, and class references are the worst offenders), but I suspect that this’ll solve your problem.

All that said, this is a cron problem (periodic recurring work), rather than a DJ problem, in my opinion. You can have a one-shot analytics parser that runs your analytics every X minutes with script/runner, invoked by cron, which very neatly cleans up any potential memory leaks or misuses per-run (since the whole process terminates at the end)