Question

I’m implementing a utility library which is a sort-of task manager intended to run within the distributed environment of Google App Engine cloud computing service. (It uses a combination of task queues and memcache to execute background processing). I plan to use generators to control the execution of tasks, essentially enforcing a non-preemptive “concurrency” via the use of yield in the user’s code.

The trivial example – processing a bunch of database entities – could be something like the following:

class EntityWorker(Worker):
    def setup():
        self.entity_query = Entity.all()
    def run():
        for e in self.entity_query:
            do_something_with(e)
            yield

As we know, yield is two way communication channel, allowing to pass values to code that uses generators. This allows to simulate a “preemptive API” such as the SLEEP call below:

def run():
    for e in self.entity_query:
        do_something_with(e)
        yield Worker.SLEEP, timedelta(seconds=1)

But this is ugly. It would be great to hide the yield within seperate function which could invoked in simple way:

self.sleep(timedelta(seconds=1))

The problem is that putting yield in function sleep turns it into a generator function. The call above would therefore just return another generator. Only after adding .next() and yield back again we would obtain previous result:

yield self.sleep(timedelta(seconds=1)).next()

which is of course even more ugly and unnecessarily verbose that before.

Hence my question: Is there a way to put yield into function without turning it into generator function but making it usable by other generators to yield values computed by it?

Answer 1

Alas, this won’t work. But a “middle-way” could be fine:

def sleepjob(*a, **k):
    if a:
        return Worker.SLEEP, a[0]
    else:
        return Worker.SLEEP, timedelta(**k)

So

yield self.sleepjob(timedelta(seconds=1))
yield self.sleepjob(seconds=1)

looks ok for me.