Question

I’m working on a project which uses Twisted to provide a high performance UDP server capable of handling burst traffic of 5k packets / second, with packet ranging in size from 50 to 100 bytes. The PC I’m testing the server on has a quad-core CPU with 4GB RAM and is running Ubuntu 10.1.

In my performance tests, I’m using tcpreplay to send previously captured traffic consisting of 500 UDP packets to the Twisted UDP server as fast as possible. The tests are between two physical (non-VM) machines on the same gigabit LAN. According to tcpreplay, I’m sending the packets at ~1250 packets / second, but out of the 500 packets that I’ve sent, only ~350-400 packets are received by the Twisted UDP server.

What kind of performance tuning can I do in Twisted or on a system level to boost performance and prevent too many dropped UDP packets?

Server Code

#!/usr/bin/env python

from twisted.internet import reactor
from twisted.internet.protocol import DatagramProtocol

packetCount = 0

class DeviceProtocol(DatagramProtocol):

    "Show me how many packets the server has received"
    def datagramReceived(self, datagram, address):
        global packetCount
        packetCount += 1
        print "Received packet %s" % packetCount

def main():
    reactor.listenUDP(7000, DeviceProtocol())
    reactor.run()

if __name__ == '__main__':
    main()

Custom Sysctl.conf Settings

net.core.netdev_max_backlog=2500
net.core.rmem_max=16777216
net.core.wmem_max=16777216

Updated Answers

• CPU Usage never goes above 10%
• Redirecting the output to a file, or only printing every 100th “Received packet” message makes a small difference, I’m still seeing 10-20% dropped packets.
• Tested that my network is stable and performs well using iptraf

Answer 1

I’ve tested my network using iperf, which is an excellent tool that I’ll be adding to my toolbox, and my network handles all of the traffic without issue.

It turns out my issues were due to:

• Using a small capture file
• Replaying the capture using tcpreplay’s –loop option

In my tests, tcpreplay drops packets if it loops too quickly over a small capture file. My capture file consisted of 100 packets, and looping over it four or five times would cause 10-20% of the packets to not be sent to the receiving side.