Question

(See this question in ServerFault)

I have a Java client that uses Socket to open concurrent connections to the same machine. I am witnessing a phenomenon where one request completes extremely fast, but the others see a delay of 100-3000 milliseconds. Packet inspection using Wireshark shows all SYN packets beyond the first wait a long time before leaving the client. I am seeing this on both Windows and Linux clients. What could be causing this? This happens when the client is a Windows 2008 or a Linux box.

Code attached:

import java.util.*;
import java.net.*;

public class Tester {
    public static void main(String[] args) throws Exception {
        if (args.length < 3) {
            usage();
            return;
        }
        final int n = Integer.parseInt(args[0]);
        final String ip = args[1];
        final int port = Integer.parseInt(args[2]);

        ExecutorService executor = Executors.newFixedThreadPool(n);

        ArrayList<Callable<Long>> tasks = new ArrayList<Callable<Long>>();
        for (int i = 0; i < n; ++i)
            tasks.add(new Callable<Long>() {
                public Long call() {
                    Date before = new Date();
                    try {
                        Socket socket = new Socket();
                        socket.connect(new InetSocketAddress(ip, port));
                    }

                    catch (Throwable e) {
                        e.printStackTrace();
                    }
                    Date after = new Date();
                    return after.getTime() - before.getTime();
                }
            });
        System.out.println("Invoking");
        List<Future<Long>> results = executor.invokeAll(tasks);
        System.out.println("Invoked");
        for (Future<Long> future : results) {
            System.out.println(future.get());
        }
        executor.shutdown();
    }

    private static void usage() {
        System.out.println("Usage: prog <threads> <url/IP Port>");
        System.out.println("Examples:");
        System.out.println("  prog tcp 10 127.0.0.1 2000");
    }
}

Update – the problem reproduces consistently if I clear the relevant ARP entry before running the test program. I’ve tried tuning the TCP retransmission timeout, but that didn’t help. Also, we ported this program to .Net, but the problem still happens.

Updated 2 – 3 seconds is the specified delay in creating new connections, from RFC 1122. I still don’t fully understand why there is a retransmission here, it should be handled by the MAC layer. Also, we reproduced the problem using netcat, so it has nothing to do with java.

Answer 1

I haven’t found a real answer from this discussion. The best theory I’ve come up with is:

1. TCP layer sends a SYN to the MAC layer. This happens from several threads.
2. First thread sees that IP has no match in the ARP table, sends an ARP request.
3. Subsequent threads see there is a pending ARP request so they drop the packet altogether. This behavior is probably implemented in the kernel of several operating systems!
4. ARP reply returns, the original SYN request from the first thread leaves the machine and a TCP connection is established.
5. TCP layer waits 3 seconds as stated in RFC 1122, then retries and succeeds.

I’ve tried tweaking the timeout in Windows 7 but wasn’t successful. If anyone can reproduce the problem and provide a workaround, I’ll be most helpful. Also, if anyone has more details on why exactly this phenomenon happens only with multiple threads, it would be interesting to hear.

I’ll try to accept this answer as I don’t think any of the answers provided a true explanation (see this discussion on meta).