Question

I’m actually working with a multi threaded program which involves lots of mysql operations, and basically it’s quite a pain as you have to come up with a smart way to make all queries work. This got me thinking that how you can make a module thread safe.

Anyway I’m trying to ask my question in this way: say you need to constantly append new content to a txt file with lots of different threads, main.py would definitely work as below:

import threading

lock = threading.RLock()

def AppendStr(the_str):
    write_thread = threading.Thread(target = RealAppending, args = (the_str, ))
    write_thread.start()

def RealAppending(the_str):
    lock.acquire()

    the_file = open("test.txt", "a")
    the_file.append(the_str)
    the_file.close()

    lock.release()

def WorkerThread(some_arg):
    do stuff

    AppendStr("whatever you like")

for counter in range(100):
    new_thread = threading.Thread(target = WorkerThread, args = (some_arg, ))
    new_thread.start()

Well, the thing is, if I’m trying to make the codes neat and easier to maintain, does it still work if I put the codes below into write.py:

import threading

lock = threading.RLock()

def AppendStr(the_str):
    write_thread = threading.Thread(target = RealAppending, args = (the_str, ))
    write_thread.start()

def RealAppending(the_str):
    lock.acquire()

    the_file = open("test.txt", "a")
    the_file.append(the_str)
    the_file.close()

    lock.release()

and do it like this in main.py: (I don’t truly understand how import works in python)

import write

def WorkerThread(some_arg):
    do stuff

    write.AppendStr("whatever you like")

for counter in range(100):
    new_thread = threading.Thread(target = WorkerThread, args = (some_arg, ))
    new_thread.start()

And also what if there are lots of other modules using write.py in a multi-threaded way, and then you import those modules in main.py and call different def from there. Would everything work out as expected? If not, what should I do to design a ultimate thread-safe module which can be used like this?

If you write.py is imported in lots of other modules, do they all share the same lock? What’s the scope of variables in such modules?

Answer 1

this looks like a threadsave module but there are mistakes:

this is a better version:

def RealAppending(the_str):
    lock.acquire()
    try:
        the_file = open("test.txt", "a")
        try:
            the_file.write(the_str) # write
        finally:
            the_file.close()
    finally: # if an error occurs
        lock.release()

because:

• if an error occurs when writing to the file the lock must be released

the syntax above is for python 2.3 and lower

here is the improved version that does exactly the same:

def RealAppending(the_str):
    with lock:
        with open("test.txt", "a") as the_file:
            the_file.write(the_str) # write

So yes, your module is threadsave.
There is some stuff that can be added to prevent users from using it in an nonthreadsave manner:

# write.py
__all__ = ['AppendStr'] # put all functions in except the ones others shall not see
# so id you do from write import * nothing else will be imported but what is __all__

But still you cannot know in which order the strings will be written to the file.