Question

I am using a Redis pubsub channel to send messages from a pool of worker processes to my ASP.NET application. When a message is received, my application forwards the message to a client’s browser with SignalR.

I found this solution to maintaining an open connection to Redis, but it doesn’t account for subscriptions when it recreates the connection.

I’m currently handling Redis pubsub messages in my Global.asax file:

public class Application : HttpApplication
{
    protected void Application_Start()
    {
        var gateway = Resolve<RedisConnectionGateway>();
        var connection = gateway.GetConnection();
        var channel = connection.GetOpenSubscriberChannel();

        channel.PatternSubscribe("workers:job-done:*", OnExecutionCompleted);
    }

    /// <summary>
    /// Handle messages received from workers through Redis.</summary>
    private static void OnExecutionCompleted(string key, byte[] message)
    {
        /* forwarded the response to the client that requested it */
    }
}

The problem occurs when the current RedisConnection is closed for whatever reason. The simplest solution the problem would be to fire an event from the RedisConnectionGateway class when the connection has been reset, and resubscribe using a new RedisSubscriberChannel. However, any messages published to the channel while the connection is being reset would be lost.

Are there any examples of recommended ways to handle this situation?

Answer 1

Yes, if the connection dies (network instability, re-mastering, whatever) then you will need to re-apply any subscriptions you have made. An event to reconnect and resubscribe is pretty normal, and not very different to what we use here on SE/SO (except we typically track more granular subscriptions, and have some wrapper code that handles all that).

Yes, any events published while your connection was broken are gone. That is the nature of redis pub/sub; it does not guarantee delivery to disconnected clients. Either use a tool that does promise this, or use redis to drive a queue instead – pushing/popping to/from opposite ends of a list is usually a reasonable alternative, and ensures nothing is lost (as long as your software doesn’t drop it after popping it from the list). If it helps, I have on my list a request to add the blocking pop methods – they totally destroy the multiplexer intent, but they have genuine use in some cases, so I’m not against adding them.