Question

My python script creates alot of threads, they are all daemon threads, I find that I get an error saying “out of memory”.

How do I kill a daemon thread whilst my script/application is running?

I understand the concept of daemon threads, that they destroy themselves when my process(script or application) closes/finishes. But I want to kill some of my daemon threads whilst my script is still running to avoid the “out of memory” error.

Will my thread below kill itself when there are no more tasks in the queue?

class ParsePageThread(threading.Thread):
    THREAD_NUM = 0

    def __init__(self, _queue):
        threading.Thread.__init__(self)
        self.queue       = _queue

    def run(self):
        while(True):
            try:
                url = self.queue.get()
            except Queue.Empty,e:
                return # WILL this kill the thread?
            finally:
                self.queue.task_done()

Answer 1

I’ll answer your second question first because it is easier. Yes, returning from the run method will indeed stop the thread. A detailed explanation is threading: Thread Objects doc.

To stop a thread that is running before it’s natural completion you have to get a little more creative. There is no direct kill method on a thread object. What you need to do is use a shared variable to define the state of the thread.

alive = True

class MyThread(threading.Thread):
    def run():
        while(alive):
            #do work here

In some other piece of code, when you detect a condition for stopping that thread, the other thread simply sets alive to False:

alive = False

This is a simple example, I’ll leave it to you to scale to multiple threads.

DANGER

This example works because reading and setting a boolean variable are atomic actions in python because of the Global Interpreter Lock. Here is an excellent tutorial for lower level python threading. You should stick to using the Queue object because that’s exactly what it’s for.

If you do anything more than reading and setting simple variables from multiple threads you should use Locks or alternatively Reentrant Locks depending on your design and needs. Even something as simple as a compare and swap without a lock can cause problems in your program that are very difficult to debug.

Another piece of advice for python multithreading is to never do any significant work in the interpreter thread. It should setup and start all the other threads and then sleep or wait on a condition object until the program exits. The reason for this is no other python thread can receive operating system signals. This means that no other thread can deal with Ctrl+C aka KeyboardInterrupt exceptions. It can be a good practice to have the main thread handle the KeyboardInterrupt exception and then set all the alive variables to False so you can exit your program quickly. This is especially helpful while developing so you don’t have to constantly kill things when you make a mistake.