I’m wondering about writing an application in Java which can synchronize files across my devices automatically (desktop, notebook, smartphone). Similar to Dropbox, however without a 3rd party server because i own my own.
Searching through the net i haven’t found any helpful topics talking about synchronization algorithms which are used in such programs. I guess it’s not an easy subject considering the fact that this problem is similar to revision control. Because of that i thinking about relaxing it. Eg. i don’t (yet) want to implement optimization in data transfer for in-file changes so it’s okay for me to transfer the whole file if it changed. Also it’s fine for me if the system ask me for resolving conflicts manually between different file versions.
My biggest problem is the network communication. How should a slave client ask/tell my central storage server about file changes? I guess it’s not a good idea to periodically transfer the whole file list with metadata to the server. As far as i see Dropbox checks the whole shared folder on startup then somehow notices changes on the local filesystem (i know there’s a way to listen to file system change events in the OS, eg. Java7 has the WatchService class for this) and also listen to network sync events. Should i maintain file revisions like version control systems doing it or is there other solutions?
Are there any books or online materials about this subject?
My first thought is rsync. There was a Java implementation.. it might work but its not longer developed: JarSync
The rsync algorithm transfers delta’s of files so you don’t incur the bandwidth hit of transferring the entire file. If you are on linux, it might be easier to execute the native rsync client from the command line.
The WatchService is a nice utility to use to monitor for changes. I believe it hooks into the operating systems native file change events rather than some naive polling mechanism that had to be used in years past.
If you want versioning support perhaps you should look into something like SVN? SVNKit is probably the best java svn client. I am not sure if SVN will transfer deltas. (I doubt it). Maybe you could look into using Git instead. I know it transfers deltas. Here is an eclipse project Git client.
Good luck!