I am writing a distributed application involving several front-end node that need to deny action to the user unless they are part of a list.
Right now we have more then 4 of those noded but only a single database server running DB2 which is often down for maintenance.
Right now we are polling the database to update an in memory list so that if a user is removed from the list the change get reflected to all 4 nodes. But if one of the node is rebooted while the database is down we would end up with an empty list which will deny all user request which we dont want. We can accept request from user even if the database is down as we buffer them in a message-queue but we want to reject them immediatly if they need to be rejected!
Does it make sense to run a Zookeeper instance on each of our 4 node and store the user permission in Zookeeper. Reading should thus be fast and the data highly available and consitent. We would not have to do polling anymore and even if we restart the database the node will be able to get their config from zookeeper!
Yes, the way you’ve described your problem, Zookeeper should fit the bill perfectly. There are few questions though that needs to be answered:
How much data are we talking about? Zookeeper persists data to disk, but works only if data fits RAM.
How often is data changed? Zookeeper will assure that more than half nodes received the update, so writes aren’t exactly performant.
How much data should be read at once? Zookeeper has limit of 1MB response size, but their recommendation is to keep data well below that limit. Note that this limit can also be reached if you are listing a node with lots of children, as children names count as data.
Considering that data is served from RAM, reading it shouldn’t be much of a problem, but you can always cache results, and set watch on appropriate nodes to invalidate local data.