Question

I have created a suffix array using the Princeton implementation. However, my basic text document is very, very large and the resulting suffix array is over 500mb in size. Is there a way to compress the suffix array?

Thanks!

Answer 1

Contrary to what is said in the previous answer, you can not only compress suffix arrays, but in fact compressing suffix trees is usually implemented by first emulating the tree using a suffix array, and then compressing that.

I am not aware of any ready-to-use Java implementation of suffix array compression and the various existing algorithms are too involved to be described here in detail. There is a paper by Navarro and Mäkinen (DOI 10.1145/1216370.1216372) which provides detailed descriptions and comparisons.

But broadly speaking, there are two general approaches:

Approach A: Reducing the size of the array directly
(see section 7.1 of the paper). This involves storing only some of the entries of the suffix array, and interpolating the missing ones when needed. The interpolation is carried out using a function (called ψ in the paper), which is itself stored in the form of a large array (but not as large as the original suffix array) and an indexed bit vector.

Approach B: The FM approach (see section 9 of the paper). Here, the suffix array is basically replaced with a relatively short array C that indicates starting positions (in the suffix array) of the main lexicographic buckets (i.e. groups of suffixes starting with the same initial character), combined with another relatively large data structure Occ that enables so called backward search. Specifically, given a search pattern p=c₁..c_m, it makes it possible to iteratively narrow the bucket for character c_m to a smaller bucket for string c_m-1c_m, and then further to the bucket for c_m-2c_m-1c_m and so forth, until the final range for the complete pattern p is found. The data structure Occ that enables this is large, but compressible using various techniques, most notably wavelet trees.

Effects on search performance
The paper cited above contains careful analyses and comparisons. But again broadly speaking, compressing the suffix array will cause the search for a pattern of length m (which can be O(m) in an uncompressed suffix array, if carefully implemented) to be delayed by a factor that depends (usually logarithmically) on the length of the entire text. Furthermore, any approach making use of wavelet trees means an additional dependence on the size of the alphabet.