I am looking into performance issues of a large C#/.NET 3.5 system that exhibits performance degradation as the number of users making requests scales up to 40-50 distinct user requests per second.
The request durations increase significantly, while CPU and I/O loads appear to stay about the same. This leads me to believe we may have problem with how shared objects in our system, which are protected using c# lock() {...} statements may be affecting concurrent access performance. Specifically, I suspect that some degree of lock convoying is occurring on frequently used shared data that is protected by critical sections (because it it read/write).
Does anyone have suggestions on how to actually diagnose if lock convoying is the problem .. or if lock contention of any kind is contributing to long request times?
Lock convoys are hard to debug in general. Does your code path have sequential lock statements either directly or in branches?
The Total # of Contentions performance counter gives a base estimate of contention in the app.
Also break open a profiler and look. You can also write some perf counters to track down the slow parts of a code path. Also make sure that locks are only being held for as long as absolutely necessary.
Also check out the Windows Performance Tools. I have found these to be extremely useful as you can track down lots of low level problems like abnormal amounts of context switching.