Question

I have a scraper set up to use delayed_job so that it runs in the background.

class Scraper
  def do_scrape
    # do some scraping stuff
  end
  handle_asynchronously :do_scrape
end

Now I can comment out the handle_asynchronously line, open the console and run the scraper just fine. It does exactly what I expect it to do.

However, when I try to fire the scrape as a delayed job, it doesn’t seem to do anything at all. Further to that, it doesn’t seem to log anything important either.

Here’s how my log looks from enqueueing a job to running rake jobs:work.

County Load (1.0ms)  SELECT "counties".* FROM "counties" WHERE "counties"."name" = 'Fermanagh' LIMIT 1
   (0.1ms)  BEGIN
  SQL (20.5ms)  INSERT INTO "delayed_jobs" ("attempts", "created_at", "failed_at", "handler", "last_error", "locked_at", "locked_by", "priority", "run_at", "updated_at") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10) RETURNING "id"  [["attempts", 0], ["created_at", Mon, 30 May 2011 21:19:25 UTC +00:00], ["failed_at", nil], ["handler", "--- 

# serialized object omitted for conciseness

nmethod_name: :refresh_listings_in_the_county_without_delay\nargs: []\n\n"], ["last_error", nil], ["locked_at", nil], ["locked_by", nil], ["priority", 0], ["run_at", Mon, 30 May 2011 21:19:25 UTC +00:00], ["updated_at", Mon, 30 May 2011 21:19:25 UTC +00:00]]
   (0.9ms)  COMMIT
  Delayed::Backend::ActiveRecord::Job Load (0.4ms)  SELECT "delayed_jobs".* FROM "delayed_jobs" WHERE (locked_by = 'host:David-Tuites-MacBook-Pro.local pid:7743' AND locked_at > '2011-05-30 17:19:32.116511') LIMIT 1
   (0.1ms)  BEGIN
  SQL (0.3ms)  DELETE FROM "delayed_jobs" WHERE "delayed_jobs"."id" = $1  [["id", 42]]
   (0.4ms)  COMMIT

As you can see, it seems to just inset a job and then delete it straight away? This scraping method should take at least a few minutes.

The worst part is, it was working perfectly last night and I can’t think of a single thing I’m doing differently. I tried fixing the gem to a previous version incase it was updated recently but doesn’t seem to have fixed the problem.

Any ideas?

Answer 1

Have you configured your delayed job to delete failed jobs? Look for the following setting in your initializer:
Delayed::Worker.destroy_failed_jobs = true

If yes then set it to false and look into the delayed_jobs table for the exception due to which it failed and debug further.