Home/ Questions/Q 8818301
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-06-14T05:04:32+00:00

I’m compressing a binary stream that’s made of packets A packet is composed of

  • 0

I’m compressing a binary stream that’s made of packets

A packet is composed of 256 32-bit integers (samples). The thing is that most integers change only a few bits from the previous integer (typically 0 – 4 bits change at most from the previous sample in the stream).

Here is an example:

3322 2222 2222 1111 1111 1110 0000 0000    BIT POSITIONS
1098 7654 3210 9817 6543 2109 8765 4321
--------------------------------------------------------
1100 1001 1110 0010 0001 0101 0110 1101    Sample 1  
               *                   * 
1100 1001 1110 1010 0001 0101 0110 0101    Sample 2     changes: bit 19, 4

1100 1001 1110 1010 0001 0101 0110 0101    Sample 3     changes: none
     *            *            *   
1100 0001 1110 1011 0001 0101 0010 0101    Sample 4     changes: bit 27, 17, 7
...

My current, lossles-compression scheme is based around nibbles. Basically I’m using a control byte where I’m encoding -using single bits- which nibbles changed from the previous sample; If there’s a change, I’ll include the modified nibbles on the compression stream, otherwise they will be reconstructed from the previous sample upon decompression.

Here is how the example stream I provided would be compressed:

Control Byte: 11111111     // all nibbles change, since this is first sample
Data:         1100 1001 1110 0010 0001 0101 0110 1101 // data for all nibbles
Control Byte: 00010001     // only nibbles 3 and 7 have changes
Data:         1010 0101    // data for nibbles 3 and 7
Control Byte: 00000000     // no nibbles are changing
Data:                      // no data is required
Control Byte: 01010010     // nibbles 1, 3 and 6 have changes
Data:         0001 1011 0010   // nibbles 1, 3 and 6
...

Using this scheme, we have a fixed overhead of 256 bytes (the control bytes), with an average, variable compressed-data length of 260 bytes (the nibbles that are changing from sample to sample). Considering the uncompressed packet is 1024 bytes in length, this is practically giving us a 50% average compression rate.

This is not bad, but my gut feeling is that a much better approach is possible. Is anybody aware of a better compression strategy that exploits the fact that very few bits change from sample to sample? Lossy compression is an alternative as long as the bit-error rate after decompression is small (less than 3%) – for this particular data stream, the numerical weight of the bit positions is irrelevant, so an error ocurring in the higher bits is of no concern at all.

Thanks everyone in advance!

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Your best bet is to use existing techniques (e.g, Lempel-Ziv-Welch; flate) or precede such a method with difference coding (probably better). With difference coding you’re replacing every byte (except the first) with the difference between that byte and the one before. Now you should get lots of zeroes, and a few small values interspersed. Huffman coding or something like LZW will compress down the string of mostly zeroes quite thoroughly.

    • 0