Question

is it possible to directly decompress, using gzip, the stdout of a command fired via subprocess.Popen ?

I’ve tried this, but it’s not working :

import subprocess
pipe = subprocess.Popen(["cat tmp.txt | gzip --stdout"], stdout=subprocess.PIPE)

import gzip
output = gzip.open(pipe.stdout)

while output.readline().rstrip():
    # Do something

Any ideas ?

Answer 1

One can pass an open file instead of a filename to Python’s gzip library by creating
a gzip.GzipFile instance directly, instead of using the helper gzip.open function. However, Python’s gzip needs a seekable file, and will fail on the stream used by subprocess.

The way to create the GzipFile instance is

output = gzip.GzipFile(fileobj=pipe.stdout)

However this won’t work, as the class needs a seekable file object. If you have no problem waiting for all the subprocess output and caching the data in memory, you can workaround this with StringIO, like in:

import StringIO, subprocess
pipe = subprocess.Popen(["cat bla3.txt | gzip --stdout"], stdout=subprocess.PIPE, shell=True)
helper = StringIO.StringIO()     
helper.write(pipe.stdout.read())
helper.seek(0)
output = gzip.GzipFile(fileobj=helper) 

If you can’t do that, you will have to copy some code from Python’s gzip.py, and handle the data and calls to the inner zlib yourself.