Home/ Questions/Q 8656121
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-12T15:12:03+00:00

I am looking for a good background job processor with following ability, Works well

  • 0

I am looking for a good background job processor with following ability,

  1. Works well with MySql
  2. Can have priorities
  3. Can easily schedule anything in background( not just emails)
  4. Ability to reinitialize the job after completion (callback would be good. I have few task/jobs that keeps on running after every minutes), even a repetitive scheduler would work
  5. Should not eat up lot of memory, (have this experience with DJ)

Few options that I am looking into Resque, DJ, Beanstalkd (haven’t explored completely)
I have my production env in Amazon EC2 (if this helps for better solution)

Please suggest me which is a good option, is there something else apart from these that people use nowadays ?

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    I’d heartily recommend sidekiq – it’s extremely flexible and it uses far less resources than Resque or DelayedJob.

    It does require redis (like Resque), but redis is valuable addition to any Rails project since it can be reused as a session store and cache. Our primary db is MySQL and we deploy to EC2 🙂 We’ve used delayed job and rescue in the past, but found them problematic and heavy on the resources they use. Sidekiq uses threads and a single sidekiq worker is as efficient as several DJ/Resque workers. Here’s an interesting part of the project’s README that I can corroborate:

    You’ll find that you might need 50 200MB resque processes to peg your
    CPU whereas one 300MB Sidekiq process will peg the same CPU and
    perform the same amount of work. Please see my blog post on Resque’s
    memory efficiency and how I was able to shrink a Carbon Five client’s
    resque processing farm from 9 machines to 1 machine.

    To sum it all up:

    1. It works fine with MySQL – not really, but it doesn’t have problems with MySQL either
    2. You can have priorities by setting up different processing queues
    3. You can easily schedule anything (and there is special ala DJ support for e-mails in particular)
    4. Not quite sure about that, we use whenever + cron for repetitive jobs
    5. You’re gonna love Sidekiq’s small memory footprint
    • 0