Question

I have some troubles understanding send (2) syscall on my linux x86 box.
Consider I established an SSH connection in my app with the other host in LAN. Then I put down the network (e.g. unplug the cable) and call the function (from my app) that sends some SSH packets trough the connection. This function inside calls send like

w = send(s->fd_out,buffer, len, 0);

In debugger I found that send returns len (i.e. w == len after the call).
How this can be if network is unreachable? When I call netstat it says my SSH connection is in state ESTABLISHED even though the network is down.

Can’t understand why send executes normally and don’t return any error (like EPIPE or ECONNRESET). May be an SSH connection lives some time after the network put down?

Thanks to all.

Answer 1

It’s due to the implementation of TCP (and ssh uses TCP). Your send() just writes to a socket, which is just a file descriptor, and return means this operation is successful. It doesn’t mean the data has been sent. A file descriptor is just some pointer with state for kernel after all. It’s implemented in the kernel to keep TCP state a bit longer before failing a session. In fact, kernel is allowed to indefinitely keep this session until you explicitly call close() or kill your process. So your data is actually buffered in kernel space for network card to deliver it later.

Here is a quick experiment you can do:
Write a server that keeps receiving messages after establishing a connection

socket();
bind();
listen();
while (1) {
    accept();
    recv();
}

Write a client establishes a connection, takes cin inputs, and send a message to server whenever you hit return.

socket();
connect();
while (1) {
    getline();
    send();
}

Be careful that you NEVER call close() in while loop on either side. Now, if you unplug your cable AFTER you’ve established a connection, send a message, reconnect again, and send another message, you will find both messages on the server side.
What you will NEVER observe is that you receive the second message before the first one. You either lose them all, or receive them in order.

Now let me explain why it behaves like this. This is the state diagram of a TCP session.
https://dl.dropbox.com/u/17011409/TCP_State.png

You can see clearly that until you explicitly call close(), the connection will always be in established state. That’s expected behavior of TCP. Establishing TCP connection is expensive, and keeping a session alive is good for performance. (That’s partially how those TCP DOS works. Attackers keep establishing connections until server runs out of resources to keep TCP state information.)

In this state, your send() will be delegated to kernel for actual sending. TCP guarantees in-order, reliable delivery, but network can lose packets at any time. So TCP HAVE TO buffer your packets, and keep trying. There are algorithms to throttle this retry, but it’s buffered for quite a very long time before it declares failure. The default time out to assume a packet loss is 3 seconds in Linux. But after a loss, TCP will retry. Then try again after certain seconds. The fact you unplugged your cable is just the same situation as a packet loss along the way to the destination. Once you plug in your cable again, a retry succeeds, and TCP will start sending remaining messages in order.

I know I must have failed to explain it thoroughly. You really need to know the details of TCP to reason about this behavior. It’s required for the properties TCP is giving you. And it’s not acceptable to expose internal implementations to programmer. (How about a send call that sometimes returns within milliseconds, and sometimes returns after 10 seconds? I bet no one will want this performance bomb in their code. The point of having a TCP library is exactly to hide this ugly nature of networks.) In fact, you even need to understand multiple RFCs and algorithms of how TCP realize in-order reliable delivery over a lossy network. Congestion control comes into the play of how long the buffer will be there as well. Wikipedia is a good starting point, but it’s a full semester’s undergraduate course if you really want to understand the details.