Question

I’m not really into multithreading so probably the question is stupid but it seems I cannot find a way to solve this problem (especially because I’m using C# and I’ve been using it for a month).

I have a dynamic number of directories (I got it from a query in the DB). Inside those queries there are a certain amount of files.

For each directory I need to use a method to transfer these files using FTP in a cuncurrent way because I have basically no limit in FTP max connections (not my word, it’s written in the specifics).

But I still need to control the max amount of files transfered per directory. So I need to count the files I’m transfering (increment/decrement).

How could I do it? Should I use something like an array and use the Monitor class?

Edit: Framework 3.5

Answer 1

You can use the Semaphore class to throttle the number of concurrent files per directory. You would probably want to have one semaphore per directory so that the number of FTP uploads per directory can be controlled independently.

public class Example
{
  public void ProcessAllFilesAsync()
  {
    var semaphores = new Dictionary<string, Semaphore>();
    foreach (string filePath in GetFiles())
    {
      string filePathCapture = filePath; // Needed to perform the closure correctly.
      string directoryPath = Path.GetDirectoryName(filePath);
      if (!semaphores.ContainsKey(directoryPath))
      {
        int allowed = NUM_OF_CONCURRENT_OPERATIONS;
        semaphores.Add(directoryPath, new Semaphore(allowed, allowed));
      }
      var semaphore = semaphores[directoryPath];
      ThreadPool.QueueUserWorkItem(
        (state) =>
        {
          semaphore.WaitOne();
          try
          {
            DoFtpOperation(filePathCapture);
          }
          finally
          {
            semaphore.Release();
          }
        }, null);
    }
  }
}