Question

Ok, so I’m interested in message queueing. I really did loads of research on this subject already. I read ‘programming windows azure’ (about Azure Queues), I read loads of tutorials and information about the Azure service bus, I watched channel 9 videos on messaging patterns, etc.

But what I just don’t understand: how would one use this in a real-life scenario? All the examples just put a single string or an object with some data in a queue and they read it from the queue on ‘the other side’. But how do you know what to do with that data? For instance: I could try to save a Customer, an Order and an Address to the database, so I put these 3 objects in a queue to read the on the other side a put them in my database. How do I know what to do with these kind of objects?

Just some questions I have:

1. What kind of data would one put in a queue. A command maybe, that is executed when it’s read, so everything in the queue has the same interface and the object itself figures out what to do?
2. Between what layers would one use queues? I was thinking about putting stuff in the queue in the service layer (after it has been validated) and reading it in the data access layer so I can put it in the database.
3. How many queues should one make? Just one queue between the layers you want to connect, or multiple queues between them (maybe for different purposes, although I can’t think of any)?
4. This form of loose coupling allows us to queue requests so they can be processed later (for example: when we’d like to restart the database). That’s cool but what if I want to READ data instead of writing it? Then the database should be online. And should I read data through a queue or could my service layer just pull the data from the data access layer and pass it to the presentation layer?

I think that were most of the questions that were buzzing around in my head. I hope anyone can clear this up for me.

Answer 1

1. Well, you can put whatever you want in there. It’s just a message, so whatever your layers can agree on will work. We put in messages that indicate when emails should be sent, when videos need to be processed, etc.
2. I wouldn’t necessarily think about queuing between layers, so much as queueing between user-facing, realtime processes (eg, web servers) and background processing. Use queueing to do things that you don’t want your web roles taking the time to do while a user is waiting.
3. You can make as many as you want, really. If the queue messages have a field that says what they mean then you could hypothetically put everything in one queue, though I wouldn’t necessarily recommend that. We’ve got about half a dozen queues: one for sending notifications, one for video processing, one for report generation requests, etc.
4. I would definitely not use a queue as data storage and retrieval. It’s just the wrong tool for that job, like trying to use TCP to store data.

Note that a single worker instance (ie, VM) can read as many queues as you would like it to. You just have to write the code that way. If you have a lot of queues that don’t see much traffic, that’s a reasonable way to keep costs down. Another alternative would be to combine several message types into a single queue. That way you only have one place to look for messages.

Alternatively, you can have many workers reading from the same queue, in order to spread the work out over them evenly.