Question

I have a design question for a multi-threaded windows service that processes messages from multiple clients.
The rules are

• Each message is to process something for an entity (with a unique id) and can be different i.e DoA, DoB, DoC etc. Entity id is in the payload of the message.
• The processing may take some time (up to few seconds).
• Messages must be processed in the order they arrive for each entity (with same id).
• Messages can however be processed for another entity concurrently (i.e as long as they are not the same entity id)
• The no of concurrent processing is configurable (generally 8)
• Messages can not be lost. If there is an error in processing a message then that message and all other messages for the same entity must be stored for future processing manually.
• The messages arrive in a transactional MSMQ queue.

How would you design the service. I have a working solution but would like to know how others would tackle this.

Answer 1

My approach would be the following:

1. Create a threadpool with your configurable number of threads.
2. Keep map of entity ids and associate each id with a queue of messages.
3. When you receive a message place it in the queue of the corresponding entity id.
4. Each thread will only look at the entity id dedicated to it (e.g. make a class that is initialized as such Service(EntityID id)).
5. Let the thread only process messages from the queue of its dedicated entity id.
6. Once all the messages are processed for the given entity id remove the id from the map and exit the loop of the thread.
7. If there is room in the threadpool, then add a new thread to deal with the next available entity id.

You’ll have to manage the messages that can’t be processed at the time, including the situations where the message processing fails. Create a backlog of messages, etc.

If you have access to a concurrent map (a lock-free/wait-free map), then you can have multiple readers and writers to the map without the need of locking or waiting. If you can’t get a concurrent map, then all the contingency will be on the map: whenever you add messages to a queue in the map or you add new entity id’s you have to lock it. The best thing to do is wrap the map in a structure that offers methods for reading and writing with appropriate locking.

I don’t think you will see any significant performance impact from locking, but if you do start seeing one I would suggest that you create your own lock-free hash map: http://www.azulsystems.com/events/javaone_2007/2007_LockFreeHash.pdf

Implementing this system will not be a rudimentary task, so take my comments as a general guideline… it’s up to the engineer to implement the ideas that apply.