Question

If I had to update 50,000 users, how would I go about it in a way that is best with a background processing library and not a N+1 issue?

I have users, membership, and points.

Memberships are related to total point values. If the membership is modified with point values I have to run through all of the users to update their proper membership. This is what I need to queue so the server isn’t hanging for 30+ minutes.

Right now I have in a controller action

def update_memberberships
  User.find_each do |user|
    user.update_membership_level! # looks for a Membership defined by x points and assigns it to user. Then Saves the user.
  end
end

This is a very expensive operation. How would I optimize for processing and in background so the post is near instantaneous from the form?

Answer 1

You seem to be after how to get this done with Resque or delayed_job. I’ll give an example with delayed_job.

To create the job, add a method to app/models/user.rb:

def self.process_x_update
  User.where("z = 1").find_each(:batch_size => 5000) do |user|
    user.x = user.y + 3
    user.save
  end
end
handle_asynchronously :process_x_update

This will update all User records where z = 1, setting user.x = user.y + 3. This will complete this in batches of 5,000, so that performance is a bit more linear.

This will cause User.process_x_update to complete very quickly. To actually process the job, you should be running rake jobs:work in the background or start a cluster of daemons with ./script/delayed_job start

One other thing: can you move this logic to one SQL statement? That way you could have one statement that’s fast and atomic. You’d still want to do this in the background as it could take some time to process. You could do something like:

def process_x_update
  User.where("z = 1").update_all("x = y + 3")
end
handle_asynchronously :process_x_update