Question

The code flow is something as below.

result = []
def Discover(myList=[]):
    for item in myList: 
        t = threading.Thread(target=myFunc, Args=[item])
        t.start()

def myFunc(item):
    result.append(item+item)

Now this will start multiple threads and in current scenario the threads does some memory intensive Tasks. Thus I want to include semaphores in this so that myList behaves as a queue and number of threads must be in a limited size. What is the better way to do that?

Answer 1

1. Never use mutable objects as default parameter value in function definition. In your case: def Discover(myList=[])

2. Use Queue.Queue instead of list to provide myList if it’s necessary to update list of “tasks” when threads are running. Or… Use multiprocessing.pool.ThreadPool in order to limit number of running threads at the same time.

3. Use Queue.Queue instead of list to provide results variable. list implementation is not thread-safe, so you probably will get many problems with it.

4. You can find some examples in other SO questions, i.e. here.

P.S. ThreadPool available in Python 2.7+

$ python
Python 2.7.1 (r271:86832, Jul 31 2011, 19:30:53) 
[GCC 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2335.15.00)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from multiprocessing.pool import ThreadPool
>>> ThreadPool
<class 'multiprocessing.pool.ThreadPool'>