Question

At scheduled times (based on business rules), we need anywhere between 10,000 – 200,000 rows from a database to be processed by a long running operation. Each row needs to be processed individually (they don’t depend on each other) and it is ok for this to happen asynchronously. The successful completion of the operation must be documented (probably to a database).

I presume the best way to do this would be using a message queue, so that is it durable and could easily scale up. Firstly, is this the best solution? If so, what is the most efficient way to put all the rows from the database onto the queue, guaranteeing that all the rows have gone to the queue?

Note: We are a .Net team using C#3.5, WCF, MSMQ and SQL 2005. I have investigated NServiceBus and would be willing to use this if recommended.

I am worried about the database being the bottleneck (updating each row to mark as processed). I also don’t know how to “transactionally send” a message.

1. How do I “transactionally send” a message? What I mean by this is: load a row from db, send message, update row in db. If the update fails, I don’t want the message to send.
2. Is this a common scenario or should I be doing it a different way?
3. My worry is that updating individual rows in the db will cause a bottleneck. Can I “transactionally send” a batch of messages, then batch update the db?

Answer 1

Yes. Message queues are designed for exactly this thing. That’s what they’re for. The message queue solution can scales to astronomical size. You’ll run out of processing capability before the queues themselves are a limiting factor.

Generally, you want concurrent separate processes do run these things. Don’t waste a single brain calorie on a threaded solution. You want as many resources as you can apply, and the OS-level allocation of resources is best for this kind of thing.

You want the simplest way that you can do the following.

1. Create the Queue.

2. Create several consumer processes that are all reading from the queue.

3. Start produce process. This will execute your query and write the queue, doing as little as possible in the process.

Each consumer process will be competing for an entry in the queue. Think of Ice Hockey face-off among idle consumers. Once a consumer grabs it’s row, it can do with it as it pleases.

When the query is empty, you have to close everything down. It’s hard (but not impossible) to close down the consumers. It’s common to have a special “all done” message that you put into the queue after the query. If you have n consumers, you put n copies of this all-done into the queue so that the consumers can all shut down cleanly.

Note that this looks a lot like a Unix pipeline. For a very good reason.

---

Edit.

1. How do I “transactionally send” a message? What I mean by this is: load a row from db, send message, update row in db. If the update fails, I don’t want the message to send.

   Do not do this. It’s a mess. Your producer sends the messages. The queue itself is perfectly reliable. Messages do not “get lost” or “somehow” fail to process. The queue can be configured to work with the file system so that the messages are persistent until consumed.

   The consumers can mark things as “done” when they’re done with them. This will slow things down a lot.

   If you want to keep “processing” status, use a data warehousing technique and don’t keep it with your transactional data.

2. Yes, this is a common scenario. Everyone tries to do too much in the producer side of this sort of thing. You rarely need all the “transactional” processing you think you need. You have to detail the end-user requirements for this. Do you really need that update? Or is it just there because it seems like you should record the processing status somewhere?

   Remember queues are very reliable. Don’t reinvent a persistent queue in the database.

   Read http://www.microsoft.com/windowsserver2003/techinfo/overview/msmqfaq.mspx on “transactional messaging”. You have lots of configuration options to guarantee the message enters the queue.

3. My worry is that updating individual rows in the db will cause a bottleneck.

   Good thing to think about. So don’t do it. The question in database updates is always “why?” If it’s “for completeness” that’s no reason at all. If it’s “for recovery” or “to prevent re-processing”, you might want to think of a better design.

   Doing inserts into a log is much, much faster and still allows you to do a where-not-exists query to locate unprocessed rows in the very unlikely event of having to recover a partially processed history.

The simplest queue is the best. You rarely need a lot of transactional funny-business. Just queue the messages up and do the work on the consumer side.