Question

I have a simple server which does on-demand mirroring. Fetching is expensive (network latency, integrity check, (re)compression, etc.). Thus, I’d like to have one lock per url being fetched, so that:

• Only one thread will be fetching a single file;
• If another thread wants one of the files currently being fetched, it will block until the file is ready.

I initially though of doing something like:

# Globals
fetching = dict()
fetch_lock = threading.Lock()

...

lock = None
do_fetch = False
with fetch_lock:
    if url in fetching:
        lock = fetching[url]
    else:
        lock = threading.Lock()
        fetching[url] = lock
        do_fetch = True

# Race condition
lock.acquire()

if do_fetch:
    ...

with fetch_lock:
    lock.release()
    if do_fetch:
        del fetching[url]

# Serve file
...

The race condition is rather minor (a thread may try to access the file which isn’t yet in the cache), but I can’t find a way to solve it.

Answer 1

To solve that issue you can grab the lock inside the critical section if you are creating it (nobody else will have it) and if the lock already exists then acquire it outside the critical section.

with fetch_lock:
    if url in fetching:
        lock = fetching[url]
    else:
        lock = threading.Lock()
        fetching[url] = lock
        do_fetch = True
        lock.acquire()
if not do_fetch:
    lock.acquire()

That will solve the race condition there. Though, what i don’t understand is why the thread that creates the lock removes it from the lock table, then you can have a thread serving the file while a newer thread is fetching it. But that’s not what you asked.