Question

I have to store two files A and B which are both very large (like 100GB). However B is likely to be similar in big parts to A so i could store A and diff(A, B). There are two interesting aspects to this problem:

1. The files are too big to be analyzed by any diff library I know of because they are in-memory
2. I don’t actually need a diff – a diff typically has inserts, edits and deletes because it is meant to be read by humans. I can get away with less information: I only need “new range of bytes” and “copy bytes from old file from arbitrary offset”.

I am currently at a loss at how to compute the delta from A to B under these conditions. Does anyone know of an algorithm for this?

Again, the problem is simple: Write an algorithm that can store the files A and B with as few bytes as possible given the fact that both are quite similar.

Additional info: Although big parts might be identical they are likely to have different offsets and be out of order. The last fact is why a conventional diff might not save much.

Answer 1

Take a look at RSYNCs algorithm, as it’s designed pretty much to do exactly this so it can efficiently copy deltas. And the algorithm is pretty well documented, as I recall.