Question

I’m hoping someone else has encountered this issue before:

Issue Description

I have Socket.IO configured to use the redis-store module, which is working great for keeping our node instances in sync when broadcasting to a room / emitting events. However I’ve been testing for failure points and it seems that when the redis instance is restarted the node servers stop receiving messages from other nodes.

Here is how I am setting up the socket.io client (redis connections are created before this bit of code)

// Setup the socket server for web clients
self.ioServer = socketIo.listen(9000);
self.ioServer.enable('browser client minification');    // send minified client
self.ioServer.set('log level', 3);

self.ioServer.set( 'store', new socketIo.RedisStore ({
        redisPub: self.redisPub, 
        redisSub: self.redisSub, 
        redisClient: self.redisStore
    })
);

In essence, the socket.io redis-store fails to continue sending events through redis to the other node servers whenever redis crashes / restarts.

Are there any special steps that I need to take for reconnecting to the redis server? I’ve spent quite a bit of time searching around to hopefully find a solution to this issue, but haven’t had any luck.

Regards,

-Ryan

Answer 1

Incase anyone is interested here is the fix I’ve implemented for the above issue:

Since the system is using cluster for the socket.io nodes, I’m catching any ‘end’ events on the redis connections (reidsPub,redisSub, etc.) and killing the forked process. On the master process I catch the exit of the child process and re-fork.

On a side note, I’ve also discovered another annoying issue with redis and socket.io’s redis-store: The redis connections will eventually timeout if no data is exchanged with redis. To solve this I’ve had to implement a simple keep alive function that runs every 15 seconds that exchanges data with redis for all the above redis connections.

Normally I would be ok using the default redis client reconnection logic, but since I have no way of knowing if the redis server actually restarted, or if it was just a timeout.