Question

I found a gem I’d like to use, but I’m having trouble getting it working. The gem is called CobWeb. It looks pretty slick.

I’ve already got redis and resque working. I’ve created a a new queue that’s meant to utilize CobWeb.

class Crawler
  @queue = :crawler_queue
  def self.perform(site_id)
    site = Site.find_by_id(site_id)
    crawler = CobWeb.new(follow_redirects: false, internal_urls: true)
    crawler.start(site.homepage)

    puts crawler # I'm ultimately interested in getting a list of urls, but at this stage, I just want to see what data I get back from the crawler.
  end

The problem is when I try to run the rake taks for the queue, I get this error. I’m not exactly sure how to fix this. Any suggestions?

rake resque:work QUEUE='*' --trace
** Invoke resque:work (first_time)
** Invoke resque:preload (first_time)
** Invoke resque:setup (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute resque:setup
** Execute resque:preload
** Invoke resque:setup 
** Execute resque:work
rake aborted!
tried to create Proc object without a block
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@RobotWatcherApp/gems/sinatra-1.3.2/lib/sinatra/base.rb:1197:in `define_method'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@RobotWatcherApp/gems/sinatra-1.3.2/lib/sinatra/base.rb:1197:in `generate_method'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@RobotWatcherApp/gems/sinatra-1.3.2/lib/sinatra/base.rb:1206:in `compile!'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@RobotWatcherApp/gems/sinatra-1.3.2/lib/sinatra/base.rb:1186:in `route'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@RobotWatcherApp/gems/sinatra-1.3.2/lib/sinatra/base.rb:1168:in `get'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@RobotWatcherApp/gems/sinatra-1.3.2/lib/sinatra/base.rb:1602:in `block (2 levels) in delegate'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@RobotWatcherApp/gems/redis-namespace-1.2.1/lib/redis/namespace.rb:257:in `method_missing'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@RobotWatcherApp/gems/resque-1.21.0/lib/resque/worker.rb:444:in `job'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@RobotWatcherApp/gems/resque-1.21.0/lib/resque/worker.rb:377:in `unregister_worker'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@RobotWatcherApp/gems/resque-1.21.0/lib/resque/worker.rb:159:in `ensure in work'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@RobotWatcherApp/gems/resque-1.21.0/lib/resque/worker.rb:159:in `work'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@RobotWatcherApp/gems/resque-1.21.0/lib/resque/tasks.rb:34:in `block (2 levels) in <top (required)>'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `call'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/task.rb:205:in `block in execute'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `each'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/task.rb:200:in `execute'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/task.rb:158:in `block in invoke_with_call_chain'
/Users/bendowney/.rvm/rubies/ruby-1.9.3-p194/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/task.rb:151:in `invoke_with_call_chain'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/task.rb:144:in `invoke'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/application.rb:116:in `invoke_task'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `block (2 levels) in top_level'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `each'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/application.rb:94:in `block in top_level'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exception_handling'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/application.rb:88:in `top_level'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/application.rb:66:in `block in run'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/application.rb:133:in `standard_exception_handling'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/lib/rake/application.rb:63:in `run'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@global/gems/rake-0.9.2.2/bin/rake:33:in `<top (required)>'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@global/bin/rake:19:in `load'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@global/bin/rake:19:in `<main>'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@RobotWatcherApp/bin/ruby_noexec_wrapper:14:in `eval'
/Users/bendowney/.rvm/gems/ruby-1.9.3-p194@RobotWatcherApp/bin/ruby_noexec_wrapper:14:in `<main>'
Tasks: TOP => resque:work

Answer 1

Can’t seem to pin this down exactly, however, I’ve just released a new version of cobweb, v0.0.64, and have created a new rails app with sample code of it running.

You can get the sample app at http://github.com/stewartmckee/cobweb_sample

The only outright thing I can see above that is wrong is that :internal_urls should be an array, eg

internal_urls: [“http://www.google.com/folder1/“, “http://www.google.com/folder2/“, “http://www.otherdomain.com/*”]

This will only allow urls with these url patterns to be processed.

Have a look at the sample site and try running that on your environment to make sure its not an environmental issue.

Stewart.