Question

I’m trying to create a resque worker that does some webscraping as a background job. The worker receives the URL for a website’s homepage, does some webscraping, and then stores the results in Redis, where it’s meant to persist for five minutes.

Does the code below look like it should accomplish what I’ve described above?

class TrialScraper
  @queue = :trial_scraper_queue
  def self.perform(homepage)
    hashed_site_data = 
      {
        :homepage =>
          {
            :url => homepage,
            :title => download_title(homepage),
            :meta_tags => download_robots_tags(homepage) 
          },
        :robots_file => download_robots_file(homepage),
      }
         p hashed_site_data #just to make sure it's working
    REDIS.setex(homepage, 60*5, hashed_site_data.to_json)
  end

Right now, this doesn’t even appear Resque’s web interface.

Any idea what I’m doing wrong? I presume that the problem lies in sending the data to REDIS, but I’m not sure.

EDIT: Just to clarify, here’s some more detail. What I would of the worker above is that it appears as a job in Resque’s web interface.

The end result of my code should really just be that some a key value pair is sent to redis (outside of Resque). But when I’m in the redis client and I list all the keys, I don’t see the key listed anywhere. But I should see the homepage listed there alongside the other keys.

Answer 1

I’ve recently written a project that writes to a Resque queue & then when the job queue is processed it writes a result out to the same Redis store, so it is possible 🙂

If you’re not seeing the job in the Resque web interface then you’re most likely not enqueueing the job?

e.g/

ok = Resque.enqueue(TrialScraper, homepage)

Hope that helps?