Question

I have a Cassandra node which is joining an existing cluster. It begins to bootstrap itself, streams a bunch of data (monitored via nodetool netstats), and then seems to finish – but it doesn’t actually join the ring. What’s the deal? How do I figure out what’s going on? There are no errors in the logs at all, it’s been plenty of time, and the Cassandra process is doing very little.

Answer 1

In this case, I turned on alllll the debug logging in /etc/cassandra/log4j-server.properties, and sifted through the reams of output until I noticed a couple of statements about OutboundTCP: attempting to connect to *host* that were appearing for certain hosts but not others. Further investigation revealed that there was limited connectivity between certain nodes in the cluster, due to a networking configuration which was my own fault: the new node could connect to only 3 out of the 5 other hosts in the cluster.

After fixing this, the node was able to transfer additional data, and I was free to pursue a life of religious fulfillment.* (Also, I ran nodetool repair all over the place out of paranoia.)