Question

I am designing a Rails app that takes in requests, uses data within the request to call a 3rd party web service, process the reply and then sends out a response to the original requestor and also issues a PUT request to yet another service.

I am trying to wrap my head around how to design this Rails app as it’s different from the canonical Rails structure.

The objects are Lists and Tasks. Each List has many Tasks, and each Task belongs to a List.

The request I would get is something like:

http://myrailsapp.heroku.com/v1/lists?id=1&from=2012-02-12&to=2012-02-14&priority=high

In this example I am requesting tasks from 2/12/2012 to 2/14/2012 with a high priority in List #1

I would then issue a 3rd party web service call like this:

http://thirdpartywebservice.com/v1/lists?id=4128&from=2012-02-12&to=2012-02-14&priority=high
As you can see some processing was done on the data (id was changed in this case)

The results are then sent back to the requestor and to another web service via PUT.

My question is, how do I set up the Rails app to handle these types of behaviors? How does the controller structure change? This looks like a good use case for queues, how do I distribute multiple concurrent requests among queues?

For one thing I don’t need data persistence (data can be discarded after the response is sent out) and also data structure design is simplified. (I don’t think I need ruby objects, simply dictionaries or hashes representing these would be lighter weight and quicker to implement)

Edit

So I broke down the work flow of the app into these components

• Parse incoming request
• Construct 3rd part web service request
• Send 3rd party request
• Enqueue a worker to process the expected response
• Process the response once it arrives
• Send the parsed result back as a response

Which of the standard ruby controllers handle each of these steps? What are the models needed besides Lists and Tasks?

Answer 1

You should still use a database because passing data to Resque is messy. Rather, you should store it in the database and then pass the id to the workers, fetch the data, commit any new data or delete the record. It’s really up to you but this method is cleaner. You can also use a push service like faye to let the user know when the processing is complete.

If you expect to have many concurrent requests, I would recommend Sidekiq as it’s less of a memory hog. Having 4-5 resque workers can already suck up about 512 MB. The controller structure should not change. Please comment on anything you need clarified and I’ll be happy to update my answer.

EDIT

You would want to use a separate database store, such as Postgres. Not sure if it’s important what models you need, but essentially this is what should be happening.

1. In your controller, create a Request object which contains the query params you want to query this 3rd party service with. Then enqueue a job to be handled by Sidekiq/Resque, let’s call this ThirdPartyRequest and pass in the id of the Request object you just created as an argument. Then render a view here showing the Request object. Let’s say that Request#response is still empty cause it hasn’t been processed yet, so let the user know it’s still processing.

2. A worker then handles your job ThirdPartyRequest. ThirdPartyRequest should then fetch the Request object and obtain the query params needed to contact the third party service. It does that then gets a Request. Update the Request object with this Request then save it.

   class ThirdPartyRequest
     def self.perform(request_id)
       request = Request.find(request_id)
   
       # contact third party service
   
       request.response = ...
       request.save
     end
   end
   

3. The user can continually refresh his page to check on his/her Request object. Once it gets updated with the response, they will know its completed. If you want the page to refresh automatically, look into faye/juggernaut/private_pub or a SaaS solution like Pusher.