Question

The actual problem

I am using Rails 3.0.4 and MongoDB through mongoid.

I am using @mailgun to deliver my emails. I send both bulk(like news-letter) and transactional(like activate account, forgot password) emails. Right now, I am using a single domain(translates to single queue on mailgun’s end) to deliver those mails. Problem arises when I have lot of bulk emails already queued up and somebody registers or requests for a new password. I want the transactional emails to be delivered before the bulk mails, but mailgun queue works on a FIFO basis.

I figured a way to mitigate this can be to use different domains(hence different queues which can be processed simultaneously) for bulk and transactional mails. In rails smtp settings are application level settings rather than request level settings. So, I figured, I will use different environments for different smtp setting.

I also have a queue system for mails and am using delayed_job to handle that. I can’t figure out a way to differentiate between bulk and transactional mails in delayed_job. So, I decided to move my queue system to either rescue+redis or beanstalked+stalker, where I can tag queues and can ask a worker to only process a particular queue.

The question

I want something which is easier to maintain, least resource hungry and can scale well.

• With delayed_job I needn’t run any other server and monitor it.
• For delayed_job I was using 256MB slice on rackspace, but redis and stalker would require another server, either resque or beanstalkd.
• I have no idea about scaling, but its 2nd month since my app launched and I have already sent 30k+ emails.

If there any alternatives to porting from delayed_job to redis or stalker, please let me know.

Update:

Seems like delayed_job also have support for named queues now, but it is not documented yet. Opening a ticket to add documentation, will update with details, once I know how to use them 🙂

Answer 1

Delayed job accepts an optional priority parameter (it’s explained down near the end of the linked page).

For example:

Delayed::Job.enqueue(MailingJob.new(params[:id]), 3)

…where 3 is the priority.

So, when en-queuing your mass-mailing, don’t specify a priority, and when en-queuing your transactional emails give them something higher. This way, your transactional emails will get entered into the @mailgun queue ahead of anything that isn’t already on its way out.

Unless your outgoing SMTP server has some kind of slow connection, it’ll probably send several hundred emails per minute, so I wouldn’t worry too much if there are, for example, 200 mass-mailings already handed off to @mailgun and you’re waiting on a transactional email; it will still send shortly thereafter.