Today I profiled one of my C# applications using the Visual Studio 2010 Performance Analyzer. Specifically, I was profiling for “Concurrency” because it seemed as though my app should have more capacity then it was demonstrating. The analysis report showed that the threads were spending ~70-80% of their time in a Synchronization state.
To be honest, I’m not sure what this means. Does this mean that the application is suffering from a live-lock condition?
For context… there are ~30+ long-running threads bound to a single AppDomain (if that matters) and some of the threads are very busy (Ex. while(true) { _waitEvent.WaitOne(0); //do stuff }).
I realize this is a fairly vague question… I guess I’m looking for some elucidation on the meaning the Synchronization state of threads. How much is too much, and why? Is ~75% really bad? Do I have too many threads? or should I just start looking in other areas?
It means that the threads were on average spending 75% of their time waiting for another thread to finish some work.
Maybe!
To clarify for readers unfamiliar with the term: a ‘deadlock’ is when two threads are both waiting for each other to finish, and therefore they wait forever. A ‘live lock’ is a situation where two threads are trying to avoid a deadlock, but due to their poor choices, spend most of their time waiting anyway. Imagine for example a table with two people, a fork and a knife. Both people wish to pick up both utensils, use them, and then put them down. Suppose I pick up the knife and you pick up the fork. If we both decide to wait for the other to put the utensil down, we are deadlocked. If we both realize that we’re about to deadlock, and I put down the knife, and you put down the fork and then I pick up the fork and you pick up the knife, we are live-locked. We can repeat this process indefinitely; we’re both working to resolve the situation, but we’re not communicating effectively enough to actually resolve it quickly.
However, my guess is that you’re not in a live-lock situation. My guess is rather that you simply have enormous contention on a small number of critical resources that can only be accessed by one thread at a time. Occam’s Razor would indicate that you should assume the simple hypothesis — lots of threads taking turns using a scarce resource — rather than the complicated hypothesis — a whole bunch of threads all trying to tell each other “no, you go first”.
Sounds awful.
Yes, it is.
Well, suppose you were trying to drive across town, and you and every other driver in the city spent 75% of your time stopped at traffic lights waiting for other drivers. You tell me: is that too much, and why? Spending an hour in traffic to drive for 15 minutes distance might be perfectly acceptable to some people and utterly unacceptable to other people. Every time I take SR 520 at rush hour I spend an hour in traffic to move a distance that should take 15 minutes; that wasn’t acceptable to me so now I take the bus.
Whether this lousy performance is acceptable to you and your customers or not is your call. Fixing performance problems is expensive. The question you should be asking is how much profit you’ll gain by taking on the expense of diagnosing and fixing the problem.
Your threads are taking four times longer than they need to. Doesn’t seem too good to me.
You almost certainly do, yes. 30 is a lot.
But that is completely the wrong technical question to ask in your situation. Asking “do I have too many threads?” is like trying to fix traffic congestion by asking “does this city have too many cars?” The right question to ask is “why are there so many traffic lights in this city where there could be freeways?” The problem isn’t the threads; the problem is that they are waiting on each other instead of driving on through to their destinations without stopping.
How on earth should we know?