TLDR: Does there exist reusable code for automatically reconnecting to a TCP server that sometimes fails?
I’m writing a server application — call it hal — that also opens some TCP connections to other servers — among them xbmc. I had originally written it so that when xbmc failed, the xbmc erlang process would stop and subsequently be restarted by its supervisor.
Apparently, this is not a good way to do persistent TCP connections in erlang. First of all, it doesn’t really work: When xbmc failed, the processs would restart too quickly, and the supervisor would shut down the entire hal program. Secondly, I am apparently not supposed to use the supervisor for this: Erlang Supervisor Strategy For Restarting Connections to Downed Hosts (As I read it, this linked question only answers “does supervisor solve this?”, and is not a duplicate of my question)
I’m thinking this sounds like a reasonably common use case, keeping a TCP connection as connected as possible even with a host that goes down now and then. Is there some OTP or other library code I should use to achieve this?
No, there’s nothing in OTP that does this for you.
I would say that given the number of times I’ve written this myself, it’s high time for a decent connection management library to be written. Particularly one that considers a recent history of connection attempts and results, can be given an endpoint address (hostname and port) and use multiple addresses returned from the hostname lookup to perform fast failover for endpoints that have multiple destinations.