Question

I have a custom protocol overlaying TCP that can be described as follows:-

Client sends a packet A to the server. The server ACKS the packet A.
Client sends a packet B.

In other words, at any point in time, there is only one unacknowledged packet. Hence, the factors that play into account for sending messages as fast as possible are:-

1. How soon can a packet arrive at the destination. This implies least amount of fragmentation
   done by TCP. If a packet can arrive in a single segment as opposed to 5 segments, the quicker the server can respond to it.

2. The unit of work done by the server for that packet. At present, i am not focused on this point, though eventually, i will touch it also.

3. Also assume, the rate of loss is negligible.

4. Nagle is disabled.

5. Typical packet sizes vary from 1KB to 3KB.

6. Bandwidth is 1Gb/sec

I am thinking that if i configure MTU equal to the biggest message size (3KB + headers), this should impact the number of messages that i can send in a second. My question is that are there any negative consequences in changing MTU. This application runs inside the LAN in a managed environment.

Alternatively, if i set the don’t fragment flag, would it be equivalent to the above change?

Answer 1

First, let’s clarify the difference between MTU and MSS. These belong to different layers of the stack (2 and 3).

TCP/IP is a quite unfortunate layered cake, both of which support fragmentation but differently, and they do not cooperate on this matter.

IP fragmentation is something that TCP is unaware of. In fact, if one of the IP fragments lost, the whole series is declared lost. Not so is for TCP: if one of the IP datagrams which are part of the same TCP stream is lost, and they were fragmented by the TCP, only retransmit of the lost parts is required.

The core reason for this mess is that a router must be able to impedance-match between two physical networks with different MTUs without understanding the higher (TCP) protocol.

Now, all modern networks support “jumbo frames” (you have to configure your NIC to be able to send jumbo frames; all modern NICs will always be able to receive frames up to 90xx bytes).

As usual with increasing MTU, it is

• not useful unless you increase MSS
• improves performance (bandwidth), and
• hurts performance (zero-load latency to first byte)

In some applications, like, for example, Gigalinx implementation og GigE vision, increasing MTU is a requirement. Over fast networks the overhead of 1500 byte MTU is intolerable.

As an architect, the thing to ask yourself is what is your application actually doing. If there is a “relevant packet size”, in sense that “until first 3kB of data received there’s nothing to do with the rest”, and you really need this tiny performance edge, increase MTU. Before doing that, consider dropping TCP altogether in favor of more ethernet-friendly protocol, and of course do not implement it yourself but choose something like ZeroMQ which works good.

Second question: Do not fragment is an IP setting. Typically useful only in routers, which are expected to match networks of different MTU. It means “discard packet unless I can relay it to the other network”. If this is sometimes the case, TCP cannot work over this layer. It will try to retransmit and fail again and again, and eventually disconnect and further behavior will depend on what application is doing. This is a typical situation on the internet, with public misconfigured wifi networks and home networks. You can sometimes browse facebook but not practically watch anything on youtube. This is why. Network administrators would never know the reason.