Question

I need to filter a large lists several times, but I’m concerned with both simplicity of code and execution efficiency. To give an example:

all_things # huge collection of all things

# inefficient but clean code
def get_clothes():
    return filter(lambda t: t.garment, allThings)

def get_hats():
    return filter(lambda t: t.headgear, get_clothes())

I’m concerned that I’m iterating over the clothes list when in fact it has already been iterated over. I also want to keep the two filter operations separate, as they belong to two different classes, and I do not want to duplicate the first lambda function in the hats class.

# efficient but duplication of code
def get_clothes():
    return filter(lambda t: t.garment, allThings)

def get_hats():
    return filter(lambda t: t.headgear and t.garment, allThings)

I have been investigating generator functions, as they seemed like the way to go, but I haven’t as yet figure out how.

Answer 1

First of all using filter/lambda combination is going to be deprecated. Current functional programming style is described in Python Functional Programming HOWTO.

Secondly, if you concerned with efficiency, rather than construct lists, you should return generators. In this case they are simple enough to use generator expressions.

def get_clothes():
    return (t for t in allThings if t.garment)

def get_hats():
    return (t for t in get_clothes() if t.headgear)

Or if you’d prefer, true generators (allegedly more pythonic):

def get_clothes():
    for t in allThings:
       if t.garment:
           yield t

def get_hats():
    for t in get_clothes():
        if t.headgear:
            yield t

If for some reason, sometimes you need list rather than iterator, you can construct list by simple casting:

hats_list = list(get_hats())

Note, that above will not construct list of clothes, thus efficiency is close to your duplicate code version.