Question

I’m going to implement a (simple) downloader application in Java as a personal exercise. It is going to run several jobs in different threads, in a way that I will be downloading a few files at the same time at all times during execution.

I want to be able to define a download rate limit that is shared between all the download jobs, but I don’t know how to do it even for a single download task. How should I go about doing this? What are the solutions I should try implementing?

Thanks.

Answer 1

I’d start with a DownloadManager that manages all downloads.

interface DownloadManager
{
    public InputStream registerDownload(InputStream stream);
}

All code that wants to take part in managed bandwidth will register it’s stream with the download manager before it starts reading from it. In it’s registerDownload() method, the manager wraps the given input stream in a ManagedBandwidthStream.

   public class ManagedBandwidthStream extends InputStream
   {
      private DownloadManagerImpl owner;

      public ManagedBandwidthStream(
            InputStream original,
            DownloadManagerImpl owner
         )
      {
         super(original);
         this.owner = owner;
      }

      public int read(byte[] b, int offset, int length)
      {
          owner.read(this, b, offset, length);
      }

      // used by DownloadManager to actually read from the stream
      int actuallyRead(byte[] b, int offset, int length)
      {
          super.read(b, offset, length);
      }

      // also override other read() methods to delegate to the read() above
   }

The stream ensures all calls to read() are directed back to the download manager.

class DownloadManagerImpl implements DownloadManager
{
   public InputStream registerDownload(InputStream in)
   {
       return new ManagedDownloadStream(in);
   }

   void read(ManagedDownloadStream source, byte[] b, int offset, int len)
   {
      // all your streams now call this method.
      // You can decide how much data to actually read.
      int allowed = getAllowedDataRead(source, len);
      int read = source.actuallyRead(b, offset, len);
      recordBytesRead(read);  // update counters for number of bytes read
   }
}

Your bandwidth allocation strategy is then about how you implement getAllowedDataRead().

A simple way of throttling the bandwidth is,
keep a counter of how many more bytes can be read in a given period (e.g. 1 second). Each call to read examines the counter and uses that to restrict the actual number of bytes read. A timer is used to reset the counter.

In practice, the allocation of bandwith amongst multiple streams can get quite complex, espeically to avoid starvation and promote fairness, but this should give you a fair start.