Question

I’d like to download, extract and iterate over a text file in Python without having to create temporary files.

basically, this pipe, but in python

curl ftp://ftp.theseed.org/genomes/SEED/SEED.fasta.gz | gunzip | processing step

Here’s my code:

def main():
    import urllib
    import gzip

    # Download SEED database
    print 'Downloading SEED Database'
    handle = urllib.urlopen('ftp://ftp.theseed.org/genomes/SEED/SEED.fasta.gz')


    with open('SEED.fasta.gz', 'wb') as out:
        while True:
            data = handle.read(1024)
            if len(data) == 0: break
            out.write(data)

    # Extract SEED database
    handle = gzip.open('SEED.fasta.gz')
    with open('SEED.fasta', 'w') as out:
        for line in handle:
            out.write(line)

    # Filter SEED database
    pass

I don’t want to use process.Popen() or anything because I want this script to be platform-independent.

The problem is that the Gzip library only accepts filenames as arguments and not handles. The reason for “piping” is that the download step only uses up ~5% CPU and it would be faster to run the extraction and processing at the same time.

---

EDIT:
This won’t work because

“Because of the way gzip compression
works, GzipFile needs to save its
position and move forwards and
backwards through the compressed file.
This doesn’t work when the “file” is a
stream of bytes coming from a remote
server; all you can do with it is
retrieve bytes one at a time, not move
back and forth through the data
stream.” – dive into python

Which is why I get the error

AttributeError: addinfourl instance has no attribute 'tell'

So how does curl url | gunzip | whatever work?

Answer 1

Just gzip.GzipFile(fileobj=handle) and you’ll be on your way — in other words, it’s not really true that “the Gzip library only accepts filenames as arguments and not handles”, you just have to use the fileobj= named argument.