Question

If the application can ensure that there would always be space in the socket’s send buffer, would blocking and non blocking send have same performance? In this scenario, is there any advantages in either approach over the other?

Answer 1

The only difference between blocking and non-blocking send is whether the kernel puts your process to sleep or returns EWOULDBLOCK. So in terms of performance, there should be no difference.

However, I doubt your implicit assumption that the send cannot block just because the send buffer has free space. Imagine an idle socket on a system that places heavy demands on memory. I would not necessarily expect the kernel to “pin” the physical pages for your send buffer; I would expect it to use that memory for something useful instead. And then when you try to send, the kernel will need to grab a free page for the send buffer; and if there are no such pages available, it might decide to return EWOULDBLOCK instead of waiting on (say) swap.

Now, that is a lot of “maybes” and “mights”, and someone more familiar with the kernel’s internals could tell me I am wrong. But even if Linux does not behave this way today, it might tomorrow; and are you 100% sure you will never run your application on anything other than Linux, ever?

So I would not write my application with such a fragile assumption. I suggest you decide whether blocking or non-blocking semantics make more sense for your own code, and do not try to game the kernel’s internals.

[Update]

I was hoping I would not have to dig in to Linux internals, but an overconfident downvoter has driven me to it.

Start with net/ipv4/tcp.c at the “new_segment” label:

new_segment:
if (!sk_stream_memory_free(sk))
    goto wait_for_sndbuf;

skb = sk_stream_alloc_skb(sk, 0, sk->sk_allocation);
if (!skb)
    goto wait_for_memory;

See how “wait_for_sndbuf” is distinct from “wait_for_memory”? That is what I am talking about.

At the “wait_for_memory” label, there is a call to sk_stream_wait_memory with a timeo value that depends on whether this is a non-blocking send. That function, in turn, either puts the process to sleep or returns EAGAIN depending on timeo.

[Update 2]

Just to be clear what question I am answering…

I interpret this question to be, “If I know that my socket’s send buffer has sufficient free space, is there any difference — performance or otherwise — between a blocking and a non-blocking send on that socket?”

The premise is certainly possible if, for example, your protocol is to send one message, and then only send a new message after receiving a reply to a previous message. In this case, you know the send buffer is always empty when you send. By getting and/or setting the POSIX-standard SO_SNDBUF socket option, you can know that your socket has adequate free space. In this case, can a blocking send behave differently from a non-blocking send?

My reading of the POSIX spec says “yes” in principle. My reading of the Linux source code says “yes” in practice. I could certainly be wrong, but it would take someone more knowledgeable about POSIX or Linux to demonstrate it, and none of them have answered this question so far.

[Final (?) update]

Here is what I believe POSIX allows/requires you to assume.

If there is adequate free space in the send buffer, then a blocking send cannot block forever. This is the same sense in which a call to calloc with adequate free virtual memory cannot block forever. Eventually the system will find the resources it needs to send your data.

(Note the same is not true when the send buffer is full, in which case a blocking send might block forever depending on what is happening at the receiving end of the socket.)

However, even when there is adequate space in the send buffer, a non-blocking send might still return EWOULDBLOCK. So if you use non-blocking sockets, your code must handle this regardless of what you know about the send buffer or anything else.