Question

I have a method like this that goes through an array to find different APIs and launch a delayed_job instance for every API found like this.

def refresh_users_list
  apis_array.each do |api| 
    api.myclass.new.delay.get_and_create_or_update_users
  end
end

I have an after_filter on users#index controller to trigger this method. This is creating many jobs to be triggered that will eventually cause too many connections problems on Heroku.

I’m wondering if there’s a way I can check for the presence of a Job in the database by each of the API that the array iterates. This would be very helpful so I can only trigger a particular refresh if that api wasn’t updated on a given time.

Any idea how to do this?

Answer 1

1. In config/application.rb, add the following

   config.autoload_paths += Dir["#{config.root}/app/jobs/**/"]
   

2. Create a new directory at app/jobs/.

3. Create a file at app/jobs/api_job.rb that looks like

   class ApiJob < Struct.new(:attr1, :attr2, :attr3)
   
     attr_accessor :token
   
     def initialize(*attrs)
   
       self.token = self.class.token(attr1, attr2, attr3)
     end
   
     def display_name
       self.class.token(attr1, attr2, attr3)
     end
   
     #
     # Class methods
     #
     def self.token(attr1, attr2, attr3)
       [name.parameterize, attr1.id, attr2.id, attr3.id].join("/")
     end
   
     def self.find_by_token(token)
       Delayed::Job.where("handler like ?", "%token: #{token}%")
     end
   end
   

   Note: You will replace attr1, attr2, and attr3 with whatever number of attributes you need (if any) to pass to the ApiJob to perform the queued task. More on how to call this in a moment

4. For each of your API’s that you queue some get_and_create_or_update_users method for you’ll create another Job. For example, if I have some Facebook api model, I might have a class at app/jobs/facebook_api_job.rb that looks like

   class FacebookApiJob < ApiJob
     def perform
       FacebookApi.new.get_and_create_or_update_users(attr1, attr2, attr3)
     end
   end
   

   Note: In your Question you did not pass any attributes to get_and_create_or_update_users. I am just showing you where you would do this if you need the job to have attributes passed to it.

Finally, wherever your refresh_users_list is defined, define something like this job_exists? method

def job_exists?(tokens)
  tokens = [tokens] if !tokens.is_a?(Array) # allows a String or Array of tokens to be passed

  tokens.each do |token|
    return true unless ApiJob.find_by_token(token).empty?
  end

  false
end

Now, within your refresh_users_list and loop, you can build new tokens and call job_exists? to check if you have queued jobs for the API. For example

# Build a token

def refresh_users_list
  apis_array.each do |api| 
    token = ApiJob.token(attr1, attr2, attr3)
    next if job_exists?(token)

    api.myclass.new.delay.get_and_create_or_update_users
  end
end

Note: Again I want to point out, you won’t be able to just drop in the code above and have it work. You must tailor it to your application and the job’s you’re running.

---

Why is this so complicated?

From my research, there’s no way to “tag” or uniquely identify a queued job through what delayed_job provides. Sure, each job has a unique :id attribute. You could store the ID values for each created job in some hash somewhere

{
  "FacebookApi": [1, 4, 12],
  "TwitterApi": [3, 193, 44],
  # ...
}

and then check corresponding hash key for an ID, but I find this limiting, and not always sufficient for the problem When you need to identify a specific job by multiple attributes like above, we must create a way to find these jobs (without loading every job into memory and looping over them to see if one matches our criteria).

How is this working?

The Struct that the ApiJob extends has a :token attribute. This token is based on the attributes passed (attr1, attr2, attr3) and is built when a new class extending ApiJob is instantiated.

The find_by_token class method simply searches the string representation of the job in the delayed_job queue for a match based on a token built using the same token class method.