Question

My C program has read (using read(2) or recv(2)) a few bytes from a TCP socket on Linux. Is it possible to push these bytes back so that subsequent read(2) and recv(2) calls (issued deeply inside a library which I don’t control) will read them again?

I know about the MSG_PEEK flag of recv(2), and I’m going to use it as a workaround if pushing back turns out to be impossible.

Answer 1

What I’m asking for seems to be impossible. I ended up calling recv() with the flag MSG_PEEK. This will make the subsequent recv() or recvmsg() call in the library read the same data.

Without any other calls I can use it to conveniently look ahead only a single byte. Let’s suppose I needed to look ahead 2 bytes. I’d call recv(fd, buf, 2, MSG_PEEK). If 1 out of the 2 bytes have already arrived, then recv would return immediately, no matter how many times I call it. I can use epoll_ctl with EPOLLIN | EPOLLET to wait for the 2nd byte. If I want to know if there was an EOF afterwards, I need EOPLLIN | EPOLLET | EPOLLRDHUP. (Please note that EPOLLHUP won’t be returned on EOF.) So by using epoll_ctl I can avoid calling recv in a busy, polling loop to read the 2nd byte.

I’ve just verified on my Linux system that I can peek about 900 kB to the socket this way by default. (SO_RECVBUF is 1 MB for me by default, decreasing it with setsockopt seems to decrease how much can be received, but not by a consistent amount. Maybe I decrease it too late?)

Even the combination of MSG_PEEK and EPOLLET is a workaround, because they still don’t let me unread arbitrary bytes to the socket. All they let me do is peeking at the bytes already arrived without consuming them.