I’d like to use the Levenshtein algorithm to compare two files in VB.NET. I know I can use an MD5 hash to determine if they’re different, but I want to know HOW MUCH different the two files are. The files I’m working with are both around 250 megs. I’ve experimented with different ways of doing this and I’ve realized I really can’t load both files into memory (all kinds of string-related issues). So I figured I’d just stream the bytes I need as I go. Fine. But the implementations that I’ve found of the Levenshtein algorithm all dimension a matrix that’s length 1 * length 2 in size, which in this case is impossible to work with. I’ve heard there’s a way to do this with just two vectors instead of the whole matrix.
How can I compute Levenshtein distance of two large files without declaring a matrix that’s the product of their file sizes?
Note that the values in each row of the Levenshtein matrix depend only on the values in the row above it. This means that you only need two one-dimensional arrays: one contains the values of the current row; the other is populated with the new values that you can compute from the current row. Then, you swap their roles (the “new” row becomes the “current” row and vice versa) and continue.
Note that this approach only lets you compute the Levenshtein distance (which seems to be what you want); it cannot tell you which operations must be done in order to transform one string into the other. There exists a very clever modification of the algorithm that lets you reconstruct the edit operations without using nm memory, but I’ve forgotten how it works.