Question

We have a .net 3.5 web app that consumes third party web services. The proxy was created by adding a web reference to their wsdl. This proxy is not compiled.

Our error logging is picking up frequent but intermittent exceptions:

An exception of type ‘System.Web.Services.Protocols.SoapHeaderException’ occurred and was caught

If I follow the url to the page that generated the exception, I can’t recreate it.

Edit: Here is most of the exception – where it bubbled up from

Message : Internal Error
Type : System.Web.Services.Protocols.SoapHeaderException, System.Web.Services, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a Source : System.Web.Services Help link : 
Actor : 
Code : http://schemas.xmlsoap.org/soap/envelope/:Client
Detail : 
Lang : 
Node : 
Role : 
SubCode : 
Data : System.Collections.ListDictionaryInternal
TargetSite : System.Object[] ReadResponse(System.Web.Services.Protocols.SoapClientMessage, System.Net.WebResponse, System.IO.Stream, Boolean)
Stack Trace :    at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message, WebResponse response, Stream responseStream, Boolean asyncCall)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[] parameters)
   at Vendor.getSearch(getSearchRequest getSearchRequest) in c:\WINDOWS\Microsoft.NET\Framework\v2.0.50727\Temporary ASP.NET Files\root\be43c34e\b09edc7e\App_WebReferences.pww-cf-q.0.cs:line 73

Edit 2: Inner exceptions:

I sometimes get the following inner exceptions logged:

Message : Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.
    Type : System.IO.IOException, mscorlib, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
    Source : System
    Help link : 
    Data : System.Collections.ListDictionaryInternal
    TargetSite : Int32 Read(Byte[], Int32, Int32)
    Stack Trace :    at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
       at System.Net.FixedSizeReader.ReadPacket(Byte[] buffer, Int32 offset, Int32 count)
       at System.Net.Security.SslState.StartReceiveBlob(Byte[] buffer, AsyncProtocolRequest asyncRequest)
       at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
       at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
       at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer, AsyncProtocolRequest asyncRequest)
       at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
       at System.Net.TlsStream.CallProcessAuthentication(Object state)
       at System.Threading.ExecutionContext.runTryCode(Object userData)
       at System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(TryCode code, CleanupCode backoutCode, Object userData)
       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
       at System.Net.TlsStream.Write(Byte[] buffer, Int32 offset, Int32 size)
       at System.Net.PooledStream.Write(Byte[] buffer, Int32 offset, Int32 size)
       at System.Net.ConnectStream.WriteHeaders(Boolean async)

And/Or:

Message : An existing connection was forcibly closed by the remote host
        Type : System.Net.Sockets.SocketException, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089
        Source : System
        Help link : 
        ErrorCode : 10054
        SocketErrorCode : ConnectionReset
        NativeErrorCode : 10054
        Data : System.Collections.ListDictionaryInternal
        TargetSite : Int32 Receive(Byte[], Int32, Int32, System.Net.Sockets.SocketFlags)
        Stack Trace :    at System.Net.Sockets.Socket.Receive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
           at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)

Update

We’re still working on it. Originally there was a route issue, which was resolved. We’re still getting the inner exception with socket errors. We had MS support involved today, and they looked at some traces and network captures. The web service host does round-robin DNS, and they may be responding on a different IP address for the syn syn/ack from one ip, and the next from a different ip. This is not good. This is likely quite specific to our situation, but perhaps it applies to others as well.

Microsoft Network Monitor and an application trace got us the information we needed.

Answer 1

The reason for this is is that while reading the beginning of the response from the server (the SOAP headers that are at the start of the response), the server instead of sending a SOAP response, closed the TCP/IP connection.

Obviously the error is not on the client side. It is most likely a server overload situation where instead of serving a response or a 500 or 503 HTTP error code it simply terminates the TCP/IP connection. There is a (very small) probability that some network equipment is causing these connection drops, but I would naturally start investigating the server side first.

I recommend using a tool like Fiddler or Wireshark to log one of these errors. Since it’s a forced TCP/IP connection close, I would look for communications with the server that end with TCP RST packet that forces the connection to close.