We have a architecture for our ERP customers,where by a customer can have multiple databases each running at different locations.Customer have a Head office database,where data from this different databases is accumulated on running basis.We have a file based approach currently,where we make files for all the database changes in a particular format and then upload the files to Head office location on running basis.At the head office there is a program running all the time.As soon as a file is uploaded on HO FTP,and Head office exe catches it.It downloads it and update the head office database based on some location id from where data is received.
This approach has been working fine from last 10-12 years,but now we have started facing issues..As number of locations has increased more then 100 for each customer making data flow of more then 4-5 lacs on daily basis.
Issue is with the Headoffice exe and databaseupdation,as number of files and amount of data to be updated/inserted is too much.
I have been searching for a proper and scalable solution for this functional issue..
May be replication or some other approach can help.
Help and suggestions are appreciated.
You could use SymmetricDS to synchronize the databases and consolidate data at the head office. It is an open source replication server that captures changes and sends them periodically over a web-based protocol to target databases. It was designed to work even when bandwidth is low and it has automatic recovery if the network is spotty. The data can be transformed and enriched, so you can add a location ID on the fly to identify the customer of the data. It’s been deployed to production to sync large numbers (in the thousands) of databases, so a lot of work has gone into scalability. The project development is also sponsored by a commercial company, JumpMind, who is interested in its long term success and provides commercial products and support for it.