Question

The problem:
I have a WCF service that I’m trying to performance test with JMeter and I’ve noticed that the service response times increase significantly when more concurrent requests are sent. The first concurrent request returns in the expected amount of time, but each subsequent request takes an increasing amount of time – so that the last request can take double the time of the first.

Here is a screenshot of a run in JMeter:

The Code:
I have stripped back the WCF service to bare minimum, so all the service method just contains a Thread.Sleep() to simulate a slightly longer running process.

[ServiceContract]
public interface IAvailabilityService
{
  [OperationContract]
  Thing GetSomething(Request request)
}    

[ServiceBehavior(
  InstanceContextMode = InstanceContextMode.PerCall, 
  AddressFilterMode = AddressFilterMode.Any)]
public class AvailabilityService : IAvailabilityService
{
  public Thing GetSomething(Request request)
  {
    Thread.Sleep(20000);
    return new Thing();
  }
}

The service is configured as follows:

<?xml version="1.0"?>
<configuration>
  <system.serviceModel>
    <services>
      <service name="MyService.AvailabilityService"
               behaviorConfiguration="DefaultServiceBehavior">
        <endpoint address="" 
                  binding="basicHttpBinding"
                  bindingConfiguration="bindingConfig"
                  contract="MyService.IAvailabilityService" />
      </service>      
    </services>
    <behaviors>
      <serviceBehaviors>
        <behavior name="DefaultServiceBehavior">      
          <serviceMetadata httpGetEnabled="true"/>      
          <serviceDebug includeExceptionDetailInFaults="false"/>
        </behavior>
      </serviceBehaviors>
    </behaviors>
    <bindings>
      <basicHttpBinding>
        <binding name="bindingConfig">
          <security mode="None">
            <transport clientCredentialType="None" />
            <message establishSecurityContext="false" />
          </security>
        </binding>
      </basicHttpBinding>
    </bindings>
  </system.serviceModel>
</configuration>

I have also tried configuring the service to run as other standard HTTP bindings. Also I have tried upping the service throttling levels on the binding configuration (though I’m sure that for 20 requests the defaults should be fine).

Service Set-up:
The service is running in IIS 6.1 on Windows Server 2008 R2 in .NET 3.5.
Though we have also seen the same behaviour on IIS 7.

Service Trace:
I have turned on tracing and this has revealed that the service always takes around 20 seconds to process each response, but the recieve bytes on connection has a staggered start time which reflects the differences in time reported by JMeter.

Does this suggest a lag in service activation by IIS? It can’t be a concurrency issue in WCF as you would expect the last thread to be 20x the execution of the first.

Thanks in advance

Iain

UPDATE
I’ve managed to get to a consistent response by increasing the number of worker processes in IIS to match the number of concurrent requests that I want the service to handle (20) – making the site a Web Garden. Though it surprises me that you need to do this in order to get a consistent level of performance from WCF.

I’ll leave the question open a couple more days in case anyone has any better ideas.

Answer 1

This was resolved by upping the number of worker processes for the app pool in IIS. As this blog post mentions, potentially long running processes in WCF have to be supported by multiple worker processes.

There is also a balancing act on the web server to ensure that the worker processes don’t exceed the machine RAM and eat into virtual memory, which can also slow processing down.