My problem doesn’t seem like a unique one. Basically I have a fairly expensive process that needs to run regularly but only if a user triggers a certain state.
For example, I have a “questionnaire” system and I analyze the results and provide recommendations from the analysis. The analysis and making the recommendations is an expensive process, sometimes 10-20s and could increase as the data set increases. By the way, I’m currently looking at ways to improve the performance of this algorithm.
My thoughts on this are to implement a state for that user that says that the analysis/recommendations are expired, poll this state and if it is expired run the expensive process. I would then set this state to expired when the user completes a questionnaire.
My current system is rails 3.0, ruby 1.9.2, and MySQL running on EC2 servers. I’m currently using DelayedJob to push this process to a background job but I haven’t implemented the state/trigger.
So in summary my question is 1) Is this even the correct architecture given the scenario? and 2) Is there any gem or implementation of this that I can use that works with Rails?
You could build a quick rake task to check for questionnaires that need analysis and queue jobs for processing with DJ. Then you can run the rake task at pre-defined intervals with Clockwork.
Because you’re on EC2, you won’t have to pay anything additional to run the lightweight Clockwork process, and your rake task can easily inherit the same environment as your application without any risk of being triggered by an HTTP request.