Question

The typical Python threadpool will have a structure like this one:

def run(self):
    while True:
        z=self.some_task_queue.get() 
        do_work(z)

So it appears that there is a continuous monitoring of the task queue.
How CPU intensive is this continuous monitoring of the task queue?

Would it be better to introduce some
sleep(few milliseconds) time to lower the CPU load?
In this way one could stop the monitoring of the task queue for some time when
all threads are busy and decrease the CPU load.

Answer 1

There is 0.0% cpu load while 1000 threads are blocked on .get() on my machine:

#!/usr/bin/env python
from __future__ import print_function
import os
import time
from threading import Thread
from Queue import Queue

try: import psutil # pip install psutil
except ImportError:
    psutil = None

def f(queue):
    while True:
        item = queue.get() # block until an item is available
        print("got %s" % (item,))
        break # end thread

# create threads
q = Queue()
threads = [Thread(target=f, args=(q,)) for _ in xrange(1000)]

# starts them
for t in threads:
    t.daemon = True # die with the program
    t.start()


# show cpu load while the threads are blocked on `queue.get()`
if psutil is None:
    print('Observe cpu load yourself (or install psutil and rerun the script)')
    time.sleep(10) # observe cpu load
else:
    p = psutil.Process(os.getpid())
    for _ in xrange(10):
        print("cpu %s%%" % (p.get_cpu_percent(interval=0),))
        time.sleep(1)


# finish threads
for i in range(len(threads)):
    q.put_nowait(i) #note: queue is unlimited so there is no reason to wait

for t in threads: t.join() # wait for completion
print('done')