Question

I’ve reduced a compression problem I am working on to the following:

You are given as input two n-length vectors of floating point values:

float64 L1, L2, ..., Ln;
float64 U1, U2, ..., Un;

Such that for all i

0.0 <= Li <= Ui <= 1.0

(By the way, n is large: ~10^9)

The algorithm takes L and U as input and uses them to generate a program.

When executed the generated program outputs an n-length vector X:

float64 X1, X2, ..., Xn;

Such that for all i:

L1 <= Xi <= Ui

The generated program can output any such X that fits these bounds.

For example a generated program could simply store L as an array and output it. (Notice this would take 64n bits of space to store L and then a little extra for the program to output it)

The goal is that the generated program (including data) as small as possible, given L and U.

For example suppose that it happens that every element of L was less than 0.3 and every element of U was greater than 0.4 than the generated program could just be:

for i in 1 to n
    output 0.35

Which would be tiny.

Can anyone suggest a strategy, algorithm or architecture to tackle this?

Answer 1

This simple heuristic is very fast and should provide very good compression if the bounds allow for a very good compression:

Prepare an arbitrary (virtual) binary search tree over all candidate values. float64s share the sorting order with signed int64s, so you can arbitrarily prefer (have nearer to the root) the values with more trailing zeroes.

• For each pair of bounds
  • start at the root.
  • While the current node is larger than both bounds OR smaller than both bounds,
    • descend down the tree.
  • append the current node into the vector.

For the tree mentioned above, this means

• For each pair of bounds
  • find the (unique) number within the specified range that has as few significant bits as possible

  . That is, find the first bit where both bounds differ; set it to 1 and all following bits to 0; if the bit that’s set to 1 is the sign bit, set it to 0 instead.

Then you can feed this to a deflateing library to compress (and build a self-extracting archive).

---

A better compression might be possible to achieve if you analyse the data and build a different binary search tree. Since the data set is very large and arrives as a stream of data, it might not be feasible, but this is one such heuristic:

• while the output is not fully defined
  • find any value that fits within the most undecided-for bounds:
    • sort all bounds together:
      • bounds with lower value sort before bounds with higher value.
      • lower bounds sort before upper bounds with the same value.
      • indistinguishable bounds are grouped together.
    • calculate the running total of open intervals.
    • pick the largest total. Either the upper or the lower bound will do. You could even try to make a “smart choice” by splitting the interval with the least amount of significant bits.
  • set this value as the output for all positions where it can be used.

Instead of recalculating the sort order, you could cache the sort order and only remove from that, or even cache the running total as well (or switch from recalculating the running total to caching the running total at runtime). This does not change the result, only improve the running time.