Question

I have a webapp that needs to sometimes download some bytes from a url and package it up and send back to the requester. The downloaded bytes are stored for a little while so they can be reused if the same url is needed to be downloaded. I am trying to figure out how best to prevent the threads from downloading the same url at the same time if the requests come in at the same time. I was thinking of creating a class like below that would prevent the same url from being downloaded at the same time. If a url is unable to be locked then it either waits until its not locked anymore to try and download it as long as it does not exist after the unlock.

public class URLDownloader
{
    HashMap<String,String> activeThreads = new HashMap<String,String>();

    public synchronized void lockURL(String url, String threadID) throws UnableToLockURLException
    {
        if(!activeThreads.containsKey(url))
            activeThreads.put(url, threadID)
        else
            throw UnableToLockURLException()
    }

    public synchonized void unlockURL(String url, String threadID)
    {
        //need to check to make sure its locked and by the passed in thread
        returns activeThreads.remove(url); 
    }

    public synchonized void isURLStillLocked(String url)
    {
        returns activeThreads.contains(url); 
    }

}

Does anyone have a better solution for this? Does my solution seem valid? Are there any open source components out there that already do this very well that I can leverage?

Thanks

Answer 1

It sounds like you don’t need a lock, since if there are multiple requests to download the same URL, the point is to download it only once.

Also, I think it would make more sense in terms of encapsulation to put the check for a stored URL / routine to store new URLs in the URLDownloader class, rather than in the calling classes. Your threads can simply call e.g. fetchURL(), and let URLDownloader handle the specifics.

So, you can implement this in two ways. If you don’t have a constant stream of download requests, the simpler way is to have only one URLDownloader thread running, and to make its fetchURL method synchronized, so that you only download one URL at a time. Otherwise, keep the pending download requests in a central LinkedHashSet<String>, which preserves order and ignores repeats.