Question

I want to ensure an authorative remote file is in sync with a local file, without necessarily re-downloading the entire file.

I did mistakenly use wget -c http://example.com/filename

If “filename” was appended to remotely, that works fine. But if filename is prepended to, e.g. “bar” is prepended to a file just containing “foo”, the end downloaded result filename contents in my test were wrongly “foo\nfoo”, instead of “bar\nfoo”.

Can anyone else suggest a different efficient http downloading tool? Something that looks at server caching headers or etags?

Answer 1

I believe that wget -N is what you are looking for. It turns on timestamping and allows wget to compare the local file timestamp with the remote timestamp. Keep in mind that you might still encounter corruption if the local file timestamp cannot be trusted e.g. if your local clock is drifting too much.