Good time of a day!
My team develop the soft to backup files to web storage. Now it has simle functionality – just take files from hdd, check date in local and remote file and update remote to new local.
The product owner want to to make backup in another way: on remote server replace only chenge peace of file. They want to make less traffic to storage (it’s Amazon S3)
Now we have a next solution: brake file into peaces (4 kb each), count checksum of each peace, store this data in local db and at next backup verify checksums and transfer only changed peace.
But the metadata take much place on db if we backup a big amount of files
Can anybody have another solution? Thanks for help!
Rsync uses rolling hash algorithm, so you might research that.