Question

I have a thread that appends rows to self.output and a loop that runs until self.done is True (or the max execution time is reached).

Is there a more efficient way to do this other than using a while loop that constantly checks to see if it’s done. The while loop causes the CPU to spike to 100% while it’s running..

time.clock()
while True:

    if len(self.output):
        yield self.output.pop(0)

    elif self.done or 15 < time.clock():
        if 15 < time.clock():
            yield "Maximum Execution Time Exceeded %s seconds" % time.clock()
        break

Answer 1

Are your threads appending to self.output here, with your main task consuming them? If so, this is a tailor-made job for Queue.Queue. Your code should become something like:

import Queue

# Initialise queue as:
queue = Queue.Queue()
Finished = object()   # Unique marker the producer will put in the queue when finished

# Consumer:
try:
    while True:
        next_item = self.queue.get(timeout=15)
        if next_item is Finished: break
        yield next_item

except Queue.Empty:
    print "Timeout exceeded"

Your producer threads add items to the queue with queue.put(item)

[Edit] The original code has a race issue when checking self.done (for example multiple items may be appended to the queue before the flag is set, causing the code to bail out at the first one). Updated with a suggestion from ΤΖΩΤΖΙΟΥ – the producer thread should instead append a special token (Finished) to the queue to indicate it is complete.

Note: If you have multiple producer threads, you’ll need a more general approach to detecting when they’re all finished. You could accomplish this with the same strategy – each thread a Finished marker and the consumer terminates when it sees num_threads markers.