Question

I’m running a WSGI server and part of the API I’m writing returns some (rather large) files along with meta-data about them. I’d like to tar/gzip the files together both to conserve bandwidth and so only one file has to be downloaded. Since WSGI lets you return an iterable object, I’d like to return an iterable that returns chunks of the tar.gz file as it’s produced.

My question is what’s a good way to tar/gzip files together in Python in a way that’s amenable to streaming the output back to the user?

EDIT:

To elaborate on my response to Oben Sonne below, I’ll have a function such as:

def iter_file(f,chunk=32768): return iter(lambda: f.read(chunk), '')

Which will let me specify a chunk size to return from the file when returning it to the WSGI server.

Then it’s a simple matter of:

return iter_file(subprocess.Popen(["tar", "-Ocz"] + files, stdout=subprocess.PIPE).stdout)

or, if I want to return a file:

return iter_file(open(filename, "rb"))

Answer 1

The bz2 module provides sequential compressing. And it seems the zlib package can compress data sequentially too. So with these modules you could:

1. tar your files (shouldn’t take that long),
2. read the archive iteratively in binary mode,
3. pass read chunks to a sequential compression function, and
4. yield the compressed output of these functions so it may be consumed iteratively by some other component (WSGI)

AFAIK Python’s tar-API does not support sequential tar’ing (correct me if I’m wrong). But if your files are so large that you really need to tar sequentially, you could use the subprocess module to run tar on the command line and read its standard output in chunks. In that case you could also use the tar command to compress your data. Then you only had to read the stdout of your subprocess and yield read chunks.