Question

I’ve got three different generators, which yields data from the web. Therefore, each iteration may take a while until it’s done.

I want to mix the calls to the generators, and thought about roundrobin (Found here).
The problem is that every call is blocked until it’s done.

Is there a way to loop through all the generators at the same time, without blocking?

Answer 1

You can do this with the iter() method on my ThreadPool class.

pool.iter() yields threaded function return values until all of the decorated+called functions finish executing. Decorate all of your async functions, call them, then loop through pool.iter() to catch the values as they happen.

Example:

import time
from threadpool import ThreadPool
pool = ThreadPool(max_threads=25, catch_returns=True)

# decorate any functions you need to aggregate
# if you're pulling a function from an outside source
# you can still say 'func = pool(func)' or 'pool(func)()
@pool
def data(ID, start):
    for i in xrange(start, start+4):
        yield ID, i
        time.sleep(1)

# each of these calls will spawn a thread and return immediately
# make sure you do either pool.finish() or pool.iter()
# otherwise your program will exit before the threads finish
data("generator 1", 5)
data("generator 2", 10)
data("generator 3", 64)

for value in pool.iter():
    # this will print the generators' return values as they yield
    print value