To best explain my goal, I will simplify the problem to the basics of my requirements. Please let me know if more details are needed for clarity.
Suppose I have 10 unique numbers (0-9) that are available for assignment. Which numbers are reserved or free is contained by the database. The goal of a running Front End web service is to successfully request a number for assignment. Once the number has been assigned to the specific node, it is reserved and no other node can have it assigned.
Keep in mind that this is to be a distributed system with no single point of failure.
The caveat that is giving me trouble is Cassandra’s notion of eventual consistency. Note that I could tune Cassandra to be fully consistent at the cost of higher latency. If that is my best (and possibly only) option, I can do it, but I’d like to preserve the concept of consistency tuning.
My thoughts on the strategy is to do the following on the node:
1) Query Cassandra to get a list of free numbers.
2) Randomly select one of the free numbers.
3) Perform a Put to Cassandra saying that this node has reserved that number.
4) Continuously query Cassandra to see which node successfully reserved the number. (Continuously request, as the read may not immediately reflect the assignment.)
5) If the returned node name is the name in which this node’s reservation was filed, then the reservation was a success.
6) If the returned node name is a different node name, it means that another node requested the number at around the same time as this one, and was given the assignment. This node must go back to step 1 and try again.
I have an odd feeling that specific circumstances will cause errors (double assignment, etc.) if I use the above strategy.
Can anyone else comment on my proposed strategy, and possibly offer their own? Thanks.
You may want to check out Apache ZooKeeper. It seems like it’s exactly what you need. You can use ZooKeeper to get new assignments, and store the existing assignments in Cassandra.