Question

According to a section in this presumably accurate book,

A common use of pipes is to read a
compressed file incrementally; that
is, without uncompressing the whole
thing at once. The following function
takes the name of a compressed file as
a parameter and returns a pipe that
uses gunzip to decompress the
contents:

 def open_gunzip(filename):
     cmd = 'gunzip -c ' + filename
     fp = os.popen(cmd)
     return fp

If you read lines from fp one at a
time, you never have to store the
uncompressed file in memory or on
disk.

Maybe I’m just interpreting this wrong, but I don’t see how this is possible. Python couldn’t have any means of pausing gunzip halfway through spitting out the results, right? I assume gunzip isn’t going to block until a line of output is read before continuing to output more lines, so some buffer has to be capturing all of this (whether inside the Python interpreter or in the OS, whether in memory or on disk), meaning the uncompressed file is being stored somewhere in full…right?

Answer 1

Your assumption is faulty. gunzip does not have to see the entire file to unzip it. Read the unzip file format. There’s a directory, with offsets to the individual components.

It’s possible to unzip a file in pieces.

“uncompressed file is being stored somewhere in full…right?”

Not necessarily. Not sure why you’re assuming it or where you read it.

All low-level I/O calls can block. The write in gunzip — when writing to a pipe — can block when the pipe buffer is full. That’s the way I/O to a pipe is defined. Pipe I/O blocks.

Check the man pages for pipe for details.

If a process attempts to read from an
empty pipe, then read(2) will
block until data is available. If
a process attempts to write to a
full pipe (see below), then write(2)
blocks until sufficient data has
been read from the pipe to allow the
write to complete. Non-blocking
I/O is possible by using the fcntl(2)
F_SETFL operation to enable the
O_NONBLOCK open file status flag.