If I have several binary strings with compressed zlib data, is there a way to efficiently combine them into a single compressed string without decompressing everything?
Example of what I have to do now:
c1 = zlib.compress("The quick brown fox jumped over the lazy dog. ")
c2 = zlib.compress("We ride at dawn! ")
c = zlib.compress(zlib.decompress(c1)+zlib.decompress(c2)) # Warning: Inefficient!
d1 = zlib.decompress(c1)
d2 = zlib.decompress(c2)
d = zlib.decompress(c)
assert d1+d2 == d # This will pass!
Example of what I want:
c1 = zlib.compress("The quick brown fox jumped over the lazy dog. ")
c2 = zlib.compress("We ride at dawn! ")
c = magic_zlib_add(c1+c2) # Magical method of combining compressed streams
d1 = zlib.decompress(c1)
d2 = zlib.decompress(c2)
d = zlib.decompress(c)
assert d1+d2 == d # This should pass!
I don’t know too much about zlib and the DEFLATE algorithm, so this may be entirely impossible from a theoretical point of view. Also, I must use use zlib; so I can’t wrap zlib and come up with my own protocol that transparently handles concatenated streams.
NOTE: I don’t really mind if the solution is not trivial in Python. I’m willing to write some C code and use ctypes in Python.
Since you don’t mind venturing into C, you can start by looking at the code for gzjoin.
Note, the gzjoin code has to decompress to find the parts that have to change when merged, but it doesn’t have to recompress. That’s not too bad because decompression is typically faster than compression.