The Problem
Once upon a time, I wrote an internal application to read data from an internal database, and then take that data and POST it to a web service. The application is very simple, using a single thread and synchronous HTTP requests.
The scope of this application has now changed and it is trying to push far more data than it was ever intended to push. If it reads 1,000 records from our internal database, it will wrap them all into a single HTTP POST, putting a heavy CPU burden on the servers that host the web service that is receiving the data. Problems also arise when the web service encounters an error while processing one of the records in the POST. The XML response does not specify which specific record failed, so I have limited visibility into the success of my request.
How I’d Like to Fix It
I am going to re-design my application to be more reliable and more considerate of the servers hosting the web service. Specifically, I’d like to have a worker that collects records from the internal database every 15 minutes and turns them into jobs. These jobs would be serialized and stored in a queue (a database table, perhaps). I’d then like to have my application process the queue using several worker threads (is this a good idea?). A thread would pop a job out of the queue and process it by making an asynchronous HTTP POST to the web service up-stream. Depending on the status of the request, the job would result in a SUCCESS, FAILURE, TIMEOUT or ABORTED. The job would be updated in the database, the process would be logged and then the worker thread would move on to the next job, becoming idle if the job queue was empty.
I’m not an architect, so I don’t know the best way to implement something like this. Here are some specific questions I have on the design.
- I’ve read some negative things about multi-threading in a .NET MVC environment. Should I avoid using multiple threads since I’m not doing anything that’s really CPU intensive?
- Quartz.NET looks like it could do a lot of cool things. Should I be looking at using Quartz.NET for something like this?
- Is my design reasonable? If not, how could it be improved?
- How would you design a system to meet the goals of the new application?
I know this is a broad question, but I hope I’ve outlined my objectives clearly. Thank you in advance.
Have you considered MSMQ for this? You push the messages on the queue, read one every N minutes and have redundancy built in if any power failures occur etc. If you’re on a load balanced environment you can post to a shared queue.
In response the questions:
I’ve read some negative things about multi-threading in a .NET MVC environment. Should I avoid using multiple threads since I’m not doing anything that’s really CPU intensive?
You’re advised not to use ThreadPools in ASP.NET so the same will apply for MVC. It can throttle your application.
Quartz.NET looks like it could do a lot of cool things. Should I be looking at using Quartz.NET for something like this?
This is a replacement for scheduling rather than queues, akin to cronjobs.
Is my design reasonable? If not, how could it be improved?
The serialization part sounds great, the SUCCESS, FAILURE, TIMEOUT or ABORTED part sound good. As mentioned, MSMQ will save you the hastle of writing redundancy and a message queue system.
How would you design a system to meet the goals of the new application?
A service, that reads from a message queue every so often and performs the action you want it to do. You could also look into SQL Server Message Broker as an alternative to MSMQ. MSMQ doesn’t have very good built in tools to manage it, so you would need to build on top of it. However it has an entire .NET assembly in the framework built in for using it.
As you’re using .NET 4 you can also benefit from parallel tasks in place of manual thread management of the HTTP sending part of your system.