Question

I have a long running task that must be guided by external code. But external code need some information about this task. This is my homebrew example:

def longtask(self):
    yield self.get_step_length(1)
    for x in self.perform_step(1):
        ...
        yield x.id

    yield self.get_step_length(2)
    for x in self.perform_step(2):
        ...
        yield x.value

...
# call site
generator = self.longtask()
step1len = generator.Next()
step1pb = ProgressBar('Step 1', step1len)
# pull only step 1 items
for index, id in itertools.izip(xrange(0, step1len), generator):
    step1pb.update(index)
    ...do something with id

step2len = generator.Next()
step2pb = ProgressBar('Step 2', step1len)
# pull only step 1 items
for index, value in itertools.izip(xrange(0, step1len), generator):
    step2pb.update(index)
    ... do something other with value

Is it right to use such a complex generator protocols in python, or I need to refactor this code?

Answer 1

I’d refactor this to returning separate generators; you can use nested functions:

def longtask(self):
    def step_generator(step):
        for x in self.perform_step(step):
            ...
            yield x.id

    yield step_length_1, step_one_generator(1)
    yield step_length_2, step_one_generator(2)

generators = self.longtask()
for counter, (steplength, stepgen) in enumerate(generators):
    ProgressBar('Step %d' % counter, steplength)
    for index, value in enumerate(stepgen):
        # ....

Now you can also use the enumerate() function to add numbers to the items; that is much more readable than zipping together an xrange() and the generator.