Question

I would like to use thread to process a streaming input.

How can make the below code for an infinite input generate for example by using itertools.count

The code below will work if:
‘for i in itertools.count():’ is replaced by ‘for i in xrange(5):’

from threading import Thread
from Queue import Queue, Empty
import itertools

def do_work(q):
  while True:
    try:
        x = q.get(block=False)
        print (x)
    except Empty:
        break

if __name__ == "__main__":
  work_queue = Queue()
  for i in itertools.count():
    work_queue.put(i)

  threads = [Thread(target=do_work, args=(work_queue,)) for i in range(8)]

  for t in threads: t.start()
  for t in threads: t.join()

Answer 1

The problem is that itertools.count generates an infinite sequence. This means the for loop will never end. You should put that in it’s own function and make it a separate thread. This way you will have the queue growing while the worker threads get data off the queue.