Question

I have a service that spawns threads.
And i may have a leak of resources in a code i am using.
I have similar code in python that uses threads

import threading

class Worker(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)
    def run(self):
        # now i am using django orm to make a query
        dataList =Mydata.objects.filter(date__isnull = True )[:chunkSize]
        print '%s - DB worker finished reading %s entrys' % (datetime.now(),len(dataList))

while True:
    myWorker = Worker()
    mwWorker.start()
    while myWorker.isalive():  # wait for worker to finish 
        do_other_work()

is it ok ?
will the threads die when they finish executing the run method ?
do i cause a leak in resources ?

Answer 1

Looking at your previous question (that you linkd in a comment) the problem is that you’re running out of file descriptors.

From the official doc:

File descriptors are small integers corresponding to a file that has been opened by the current process. For example, standard input is usually file descriptor 0, standard output is 1, and standard error is 2. Further files opened by a process will then be assigned 3, 4, 5, and so forth. The name “file descriptor” is slightly deceptive; on Unix platforms, sockets and pipes are also referenced by file descriptors.

Now I’m guessing, but it could be that you’re doing something like:

class Wroker(threading.Thread):
    def __init__(self):
        threading.Thread.__init__(self)
    def run(self):
        my_file = open('example.txt')
        # operations
        my_file.close()   # without this line!

You need to close your files!

You’re probably starting many threads and each one of them is opening but not closing a file, this way after some time you don’t have more “small integers” to assign for opening a new file.

Also note that in the #operations part anything could happen, if an exception is thrown the file will not be close unless wrapped in a try/finally statement.

There’s a better way for dealing with files: the with statement:

with open('example.txt') as my_file:
     # bunch of operations with the file
# other operations for which you don't need the file