Question

I would like to be able to compare a binary file X to a directory of other binary files and find which other file is most similar to X. The nature of the data is such that identical chunks will exist between files, but possibly shifted in location. The files are all 1MB in size, and there are about 200 of them. I would like to be have something quick enough to analyze these in a few minutes or less on a modern desktop computer.
I’ve googled a bit and found a few different binary diff utilities, but none of them seem appropriate for my application.

For example there is bsdiff, which looks like it creates some a patch file which is optimized for size. Or vbindiff which just displays the differences graphically, but those don’t really seem to help me figure out if one file is more similar to X than another file.

If there is not a tool that I can use directly for this purpose, is there a good library someone could recommend for writing my own utility? Python would be preferable, but I’m flexible.

Answer 1

Here’s a simple perl script which more or less tries to do exactly that.

Edit: Also have a look at the following stackoverflow thread.