I’m writing an application that has a number (hundreds) of concurrent network operations running in parallel between two instances. Since the average lifetime of a connection is quite short (at most seconds), I thought that the overhead of using many TCP connections and doing a handshake every time (especially for a TLS handshake) would be too large.
I started to look at a couple of protocols and libraries implementing multiplexing (mostly AMQP implementations like Apache Qupid, RabbitMQ, as mentioned in answers to this question). However all of them seem to run over TCP, which introduces some overhead and doesn’t make a lot of sense (this post explains the problem quite well and comes to the conclusion that TCP multiplexing is stupid). Also all of them feel quite fat, I’d prefer something small and light (ZeroMQ unfortunately doesn’t implement multiplexing afaik). That got me thinking whether using UDP would be an option. Of course one has to implement stuff like recovery and ACKs properly, but with the knowledge about the multiple streams over the connection that should be much more efficient than simply using TCP.
Do you think that my reasoning above is correct, or did I miss something important? Are there any good C/C++ libraries out there that implement multiplexing via UDP?
Do the simplest thing that could possibly work, and make it more complex only as necessary:
use a single TCP connection and multiplex logical sessions over it
if you have multiple concurrent components in each instance which really need their own queues with pushback to throttle over-eager senders:
only if you hit some case where your logical sessions really interact badly with TCP, then consider implementing your own reliable-flow-controlled-datagram protocol