Question

We have a production system that gathers telemetry data from remote devices. This data is sent at a reasonable frequency and we end up receiving up to thousands of messages a second at peak times. The payload is circa 80 bytes per message. I am starting to do some performance testing of various storage machanisms, but I thought first of all I would try and see how fast I could push UDP without any data storage involved. I am getting approx 70,000 messages a second max throughput testing on my local machine (seems to be around the same if I use another machine to send the test data). From my rough calcuations, this is way lower than I expected given the network link capacity. The sender sits in a tight loop sending data. I am fully aware of all the issues with UDP re; lost packets, etc. I just want to get an idea of our systems weak points.

Is the throughput so low because of the small packet size?

Matt

private IPEndPoint _receiveEndpoint = new IPEndPoint(IPAddress.Any, _receivePort);
private Stopwatch sw = new Stopwatch();
private int _recievedCount = 0;
private long _lastCount = 0;
private Thread _receiverThread;
private bool _running = true;

_clientReceive = new UdpClient();
_clientReceive.Client.Bind(_receiveEndpoint);
_receiverThread = new Thread(DoReceive);
_receiverThread.Start();

  while (_running)
  {
    Byte[] receiveBytes = _clientReceive.Receive(ref _receiveEndpoint);

    _clientReceive.Receive(ref _receiveEndpoint);
    if (!sw.IsRunning)
      sw.Start();
    string receiveString = Encoding.ASCII.GetString(receiveBytes);
    _recievedCount = ++_recievedCount;
    long howLong = sw.ElapsedMilliseconds;
    if (howLong/1000 > _lastCount)
    {
      _lastCount = howLong/1000;
      Invoke(new MethodInvoker(() => { Text = _recievedCount + " iterations in " + sw.ElapsedMilliseconds + " msecs"; }));
    }
}

Answer 1

Lots of small UDP packets are definitely going to result in a lower network throughput than you’d get with larger packets however did you include the IP & UDP header sizes in your calculations?

Apart from that 70k messages/second is very very high and definitely not something you’d want to have happening across the internet if that was where the app is eventually going to be deployed. Even thousands of messages/second is high and if it was me I’d be looking to try and make the communications from the telemetry equipment less chatty perhaps by bundling up multiple readings into a single transmission.

If that’s not an option and you are on a private network and you need to increase the network throughput you may have to start looking at your network card, its driver and then fine tuning some Windows networking parameters. But whatever you do with the messages you are almost certainly going to bottle-neck on whatever processing you do on them, especially if it involves disk, way before you get to 70k messages/second (I’d be suprised if you can even get to 10K/second when you’re doing anything useful with them).