Question

I’m looking for some advice on how to optimise the following process:

App reads csv file.
For each line in the file an XML message is created
Each XML message is posted to a URL via a HTTPWebRequest

This process was designed to handle low volumes of messages (up to about 200 at a time), un suprisingly thinks have changed and it is now expected to handle up to about 3000 at a time.

The code used to post the message is here:

Public Function PostXml(ByVal XML As String) As HttpStatusCode

        Try
            Dim Bytes As Byte() = Me.Encoding.GetBytes(XML)
            Dim HTTPRequest As HttpWebRequest = DirectCast(WebRequest.Create(Me.PostURL), HttpWebRequest)

            With HTTPRequest
                .Method = "POST"
                .ContentLength = Bytes.Length
                .ContentType = "text/xml"
                .Credentials = New NetworkCredential(_Settings.NTSPostUsernameCurrent, _Settings.NTSPostPasswordCurrent)
            End With

            Using RequestStream As Stream = HTTPRequest.GetRequestStream()
                RequestStream.Write(Bytes, 0, Bytes.Length)
                RequestStream.Close()
            End Using

            Using Response As HttpWebResponse = DirectCast(HTTPRequest.GetResponse(), HttpWebResponse)
                Return Response.StatusCode
            End Using

        Catch ex As WebException

            If ex.Message.Contains("(500) Internal Server Error") Then
                Return HttpStatusCode.InternalServerError
            Else
                Throw
            End If
        End Try

Can this be optimised in terms of caching the connection used? At the moment there is a noticable delay at the line:
Using Response As HttpWebResponse while the connection is made.
Is there a way of caching this so the same connection is used for all 3000 messages rather than a new connection being created for each message?

Any advice gratefully recieved.

**Update. Thanks for the responses. To clarify, I am currently restricted to sending multiple messages due to restrictions elsewhere in the system. There is a noticable delay in responding to the request at the other end (the receiver) but this is outside my control. I am trying to ensure that the process of sending is as efficient as possible (external factors notwithstanding).

Answer 1

.NET already has connection caching… if you weren’t disposing of the response, you’d see that pretty quickly 🙂 (Just to clarify, you’re doing the right thing here. A bug I’ve seen quite often is not to have a Using statement… which causes a problem precisely because of connection caching.)

I suspect it’s not a case of making the connection, but making the request – in other words, the time is spent in areas outside your control.

I suggest you use Wireshark or Fiddler to work out where the time’s actually going – might it not just be the web service itself? (Or whatever you’re talking to.)

Another option is to use multiple threads to speed this up – but at that point, don’t forget to increase the number of connections per host (in the connectionSettings part of app.config, IIRC).