Question

I’ve got several megabytes of data like this:

11  2  1
 4  3  1
11  2  1
 4  3  1
11  2  1
 4  3  1
18  3  2

I’d like to compress it by adding lines saying “previous n lines repeated m times.” The algorithm should read lines and delay printing them out until finding the longest possible m*n, but can assume n<=10. What would be the best way to do this?

I was thinking of just keeping 10 arrays of 1..10 previous lines with repeat counters, rotating the array contents as new lines come in and printing the above message when a newly read line didn’t match the oldest entry in any of the arrays, and at least one of the arrays is filled with repeats.

Answer 1

“copy previous n lines repeated m times” is a limited version of “copy k lines starting from j lines back”. The first one is the second one with k = n * m and j = n. The more general k,j version is LZ77. (Though normally it’s bytes and not lines.)

LZ77 algorithms would work very well for this. The hash table approach used by gzip, zlib, etc., is fast and easy to code. First, define the minimum value of k (mink) that you would consider worthwhile, and define how far back you want to look for matches, i.e. the maximum value of j (maxj). Then construct a sliding window of maxj lines to search in.

As each line comes in, update a hash that depends only on the last mink lines. Look in the hash table for the last line that matches that hash, and then compare your lines directly with what’s in the sliding window there until they don’t match. Then if the resulting length is mink or more, you have a match, which is composed of a length and a distance (k and j).

Use lazy-matching, where you defer the emission of a match until you process the next line, which may give a longer match.