I have a table in a central database that gets constantly appended to. I want to batch these appends every couple of minutes and have them sent to a bunch of “slave” servers. These servers will, in turn, process that data and then discard it (think distributed warehousing). Each “slave” server needs a different subset of the data and there is a natural partitioning key I can use for that.
Basically, I need this to be eventually consistent : every batch of data to be eventually delivered to every “slave” server (reliability), even if the “slave” was down at the moment the batch is ready to be delivered (durability). I don’t care about the order in which the batches are delivered.
Possible solutions I have considered,
- MySQL replication does not fit my requirement because I would have to replicate the whole table on each server.
- ETL & ESB products are too bloated for this, I am not doing any data processing.
- Plain JMS, I could use but I’m looking for something even simpler
- JGroups is interesting but members that are left the group will not get the messages once they rejoin.
- Pushing files & ack files across servers : can do but I don’t know of any framework so would need to write my own.
Note : This question is about how to move the data from the central server to the N others with reliability & durability; not how to create or ingest it.
(Edited on Aug 24 to add durability requirement)
I ended up finding “Spring integration” which includes plugins to poll directories via SFTP for example.
http://www.springsource.org/spring-integration