Question

I have an API that services a web-based plugin for processing email. The API is responsible for two things:

1. Creating SessionIDs so the plugin can setup a dynamic link; and
2. Once an email is sent, for receiving that SessionID, the email recipients and subject line, to store the information into a new session.

Imagine the scenario where the plugin sends a request to the API:

PUT http://server.com/api/email/update/<SessionID> -d "to=<address1,address2>&subject=<subject>"

In testing this works fine: the data is saved normally. However, the plugin can’t help but send that request several times a second, bombarding my server with identical requests. The result is that I get my EmailSession object saving multiple copies of the recipients.

In terms of my database schema, I have an EmailSession model, which has_many EmailRecipients.

Here’s the relevant part of the update method in my API’s controller:

@email_session = EmailSession.find_or_create_by_session_id(:session_id => params[:id], :user_id => @user.id)

if opts[:params][:cm_to].blank? == false

    self.email_recipients.destroy_all                   

    unless opts[:params][:cm_to].blank?
        opts[:params][:cm_to].strip.split(",").each do |t|
            self.email_recipients << EmailRecipient.create(:recipient_email => t)
        end
    end
end

Admittedly, the “find_or_create” dynamic method is new to me, and I wonder if there’s something about that screwing up the works.

The symptoms I’m seeing include:

• ActiveRecord errors complaining about attempts to save a non-unique key into the database (I have an index on the SessionId)

• Duplicate recipients ending up in the EmailRecipients collection

• In the case of multiple users employing the plugin, I get recipients from other emails ending up in the wrong email session collections.

I’ve attempted to employ delayed_job to attempt to serialize these requests somehow. I haven’t had much luck with it thanks to various bugs in the current release. But I’m wondering if there’s a more fundamental problem with my approach to this solution? Any help would be appreciated.

Answer 1

I’m still not sure I understand what you’re doing, but here’s my advice.

First off I don’t think you are using find_or_create_by properly. This method has slightly confusing semantics (which is why 3.2 introduces some clearer alternatives) but as it stands it isn’t using the user_id to find the record (although it is setting user_id if a record is created). I don’t think this is what you wanted. Instead use find_or_create_by_session_id_and_user_id

This can still raise a duplicate key error since in between find_or_create checking and it creating the record there is time for someone else to create the record. If you weren’t doing anything other than creating email session rows the  rescuing this duplicate key error and then retrying should take of that: on the retry you’ll find the row that blocked your insert.

However when you then go on to add recipients you still have a potential issue because 2 things could be trying to remove recipients and add them to the same email session at the same time. This might be a good usecase for pessimistic locking. 

begin
  EmailSession.transaction do
    session = EmailSession.lock(true).find_or_create_by_bla_bla(...)
    # use the session object here, add recipients etc.
  end
rescue ActiveRecord::StatementInvalid => e
end

What is happening here is that when the email session is retrieved from the db, the row is locked (even if it doesn’t exist yet – effectively you can lock the gap where the record would go). This means that anyone else wanting to add recipients or do any other manipulation has to wait for the lock to be released. Locks last as long as the transaction in which they occur lasts  so all your work should happen in here (even if in the second part you are not actually changing the email session object any more).

You may end up with deadlocks – I don’t know what else is going on in your app but you should be prepared for them if you are using transactions. That’s what the rescue block is for: if the error message looks like a deadlock then you should probably retry some limited number of times.

Locks are (at least on MySQL) row level locks: as long as you have an index on session_id,user_id then just because one of your instance has one email session object locked doesn’t stop another instance from using another one.