Question

I have one worker role that throws data into around 10 queues that need to be processed. There is a lot of data – probably around 10-100 messages a second that gets queued up in various queues.

The queues hold different data and process them separately. There is a single queue in particular that is very active.

The way I have it setup now, I a separate worker role that spawns 10 different threads, each thread executes a method that has a while(true){get message from queue and process it}. Whenever data in the queue gets backed up we simply launch more of these processes to help speed up the processing of the data from the queue. Also, since one queue is more active, I actually launch a number of threads pointing at the same method to process data from that queue.

However, I am seeing high CPU utilization of the deployment. Almost at or near 100% constantly.

I am wondering if this is because of thread starvation? Or because accessing the queue is RESTful and the threads end up blocking each other via doing the connection and slowing things down? Or, is it because I use:

while(true)
{
   var message = get message from queue;
   if(message != null)
   {
       //process message
   }
}

And that gets executed too fast?

Every processing of the message also saves it to the Azure Table Storage or the DB – so it might be the process of saving this data that is eating up the CPU.

In effect, it’s been really hard to debug the high CPU load. So, my question is: are there general architecture changes that I can make that will help alleviate + prevent any possible issue that there might be? (e.g. instead of using while(true) using a different type of polling – although I’d imagine it’s the same in the end for that example).

Maybe simply spawning new threads using new Thread() is not the best way to go.

Answer 1

I would suggest putting a sleep statement in your loop… not only is that tight loop probably hogging CPU resources, but you also pay for storage transactions. Every ten thousand times you check the queue, it costs a penny. That’s a small cost, but it could add up over time to be significant.

I’ve also often used code like this:

while(true)
{
var msg = q1.GetMessage();
if (msg != null) { ... }
msg = q2.GetMessage();
if (msg != null) { ... }
}

In other words, poll the queues serially instead of parallelly (that should totally be a word). That way you’re only actually doing one thing at a time (useful if your tasks are CPU-instensive), but you’re still checking all the queues in each loop.