Question

Ok I’m doing some testing with Stalker and Beanstalkd. My objective is to offload 500kb post requests to queue and process them asynchronously.

So far in my testing I have this very simple example.

#worker.rb
require 'stalker'
include Stalker

job 'hello' do |args|
  puts "hi"
  sleep 1
  puts "hello"
end

and this file for adding to queue

# stalker.rb
require 'rubygems'
require 'stalker'

10.times do 
    Stalker.enqueue('hello')
    puts 'queued'
end

So in one terminal I run

$ stalk worker.rb
Working 1 jobs: [ hello ]

Then I run the stalker file

$ ruby stalker.rb

stalker executes almost instantly as expected so no blocking on that.

but the worker takes approximately 10 seconds to run. Really I want this to be much closer to 1 second for those 10 jobs because I want them to run in parallel.

Any recommendations on the next best step to do this?

UPDATE:
I’ve realised I can run multiple workers from different terminals and it will process the queue much faster e.g. 2 workers will do the process in roughly half the time.

Answer 1

As a heads up, you may want to check out the new gem I created called Backburner which is a more modern way to manage asynchronous jobs with beanstalkd in ruby. It follows a cleaner, resque-esque interface, has a truly asynchronous worker (that uses forking and multiple threads for much faster throughput) and has much better error handling and retry support. I was using stalker for 2 years before writing backburner because I wanted a better solution.

It gives you support for things like:

SomeObject.async(:pri => 1000, :delay => 2).some_method(1, 2, 3)

with the method then being automatically enqueued onto beanstalkd and processed asynchronously by backburner workers.