Question

I currently use FixedThreadPool to download images from the web, like this:

ExecutorService mThreadPool = Executors.newFixedThreadPool(MAX_THREADS);

And then I just submit new Runnables with image URL which is either downloading image from URL or if it exists in cache loads it from there.

I want to be able to ensure that only one thread at time can process specific URL (to prevent sitiuation where image is downloaded MAX_THREADS times), and if that thread thread finishes and downloads that image I want to allow next one (or all, with the same URL) to run, to load previously downloaded image from my cache.

Here’s what I mean shown on simple (I hope so) scheme: http://i43.tinypic.com/xnz3f9.jpg

I have seen some examples with custom implementation of Runnable with Queue of tasks, but all of them needed that I knew all URLs before executing these tasks, I want to use it in ListView with dynamically loaded content so that option won’t be possible.

Thanks for any help. 🙂

Answer 1

You’re going to have to implement some sort of synchronization scheme (locking) to prevent a second thread from starting to download the same file another thread is already downloading.

One solution that comes to mind would be to pass each of your Runnables constructor a reference to a Map<String,Lock> and a ReentrantLock for synchronizing access to it. Put a the filename as a key in the map while you’re downloading it along with another ReentrantLock for the other threads will wait on as the value.

You lock and check the Map before going to download a file.

If there’s not an entry there, you create a new ReentrantLock and insert into the Map. You then unlock the lock for the Map itself. When you’re done downloading the file you once again lock the map, remove the lock from the Map and unlock it, then unlock the map.

If there is an entry there, you know another thread is downloading the file and you have a lock to work with. Unlock the map, lock on the file lock, and wait until you get the lock. When you get the lock, you know the other thread is done.

Example:

...
mapLock.lock();
Lock fileLock = fileMap.get(fileName);
if (fileLock == null)
{
    fileLock = new ReentrantLock();
    fileLock.lock();
    fileMap.put(fileName, fileLock);
    mapLock.unlock();    

    // download and deal with file

    mapLock.lock();
    fileMap.remove(fileName);
    fileLock.unlock();
    map.unlock();
}
else // someone is downloading this file!
{
    mapLock.unlock();
    fileLock.lock();
    fileLock.unlock(); 

    // When you get here, you know the other thread has downloaded the file
    // do whatever it is you need to do in that case
}

A little more elegant solution would be to use a Condition along with the ReentrantLock (I’m not including getters for the sake of brevity)

public class LockSet {
    public Lock lock;
    public Condition condition;

    public LockSet() {
        lock = new ReentrantLock();
        condition = lock.newCondition();
    }
}

Now with this handy class, you can do the following:

mapLock.lock();
LockSet lockSet = fileMap.get(fileName);
if (lockSet == null)
{
    lockSet = new LockSet();
    fileMap.put(fileName, lockSet);
    mapLock.unlock();

    // download and deal with file

    mapLock.lock();
    fileMap.remove(fileName);
    lockSet.lock.lock();
    lockSet.condition.signalAll();
    mapLock.unlock();
}
else // someone is downloading
{
    lockSet.lock.lock();
    mapLock.unlock();
    lockSet.condition.await();

    // once we get here, the file has finished downloading in the other thread
}

Edit: An answer that was deleted by its author got me thinking a bit about this.

One downside to this scheme is that the threads in your pool will occasionally be waiting. Since you’re using a fixed pool size this could cause a bottleneck condition if you had a number of requests for the same file all at once. If you were to implement a queuing mechanism for tasks as you mentioned in your post, you could actually just get away with just the Map itself (Map<String,String>) without the double locking mechanism.

Your worker threads would pull the file info off the queue, check the map to see if it exists (still locking/unlocking the map lock), but put the file back in the queue in the case that someone else is downloading it (indicated solely by the fact that an entry exists for that file in the Map – you can just use null for the values)