Question

Let’s imagine I have this class :

public class FileData
{
   public Weigth { get; set; } // Not the file size, but a business weight
   public Name { get; set; } // Name of the file
   public FullPath { get; set; } // full path of the file
}

I want to explore a bunch of separate folders, from various network location. The idea is to be able, given a specific file name, to retrieve all candidate files, ordered by the weight of the file.

I want to split the exploration of each folder in a separate thread. in this case, my “store” should be concurrent access aware.

I’ve tried to use a ConcurrentDictionary<string, SortedSet<CdlFileInfo>> class to store the result of the exploration.

However, I’m a bit struggling with the correct way of populating the inner sortedset.

I’ve tried this :

class Program 
{
    private readonly static ConcurrentDictionary<string, SortedSet<CdlFileInfo>> g_Files = new ConcurrentDictionary<string,SortedSet<CdlFileInfo>>();
    public static void Main()
    {
        PopulateFileList();

        // Do something with the list of files
    }
    private static void PopulateFileList()
    {
        var sources = AnyMethodToGetFoldersList(); // IEnumarable<string>
        sources.AsParallel().ForAll(x =>
        {
            Console.WriteLine("Enumerating files in {0}", x.Folder);
            var allFiles = Directory.GetFiles(x.Folder, "*.*", SearchOption.AllDirectories);

            foreach (var file in allFiles)
            {
                var fd = new FileData {
                    Weigth = GetWeigth(file), // returns a int... the method is not important
                    Name = Path.GetFileName(file),
                    FullPath = file
                };

                            // Here is the key piece of my code

                g_Files.AddOrUpdate(
                    fileName,
                    new SortedSet<FileData>() { fd },
                    (filePath, source) =>
                    {
                        g_Files[fileName].Add(fd);
                        return g_Files[fileName];
                    }
                );
            }
        });
    }

    public class FileData
    {
       public Weigth { get; set; } // Not the file size, but a business weight
       public Name { get; set; } // Name of the file
       public FullPath { get; set; } // full path of the file
    }
    public class FileDataWeightComparer : IComparer<CdlFileInfo>
    {
        public int Compare(FileData x, FileData y)
        {
            return Comparer<int>.Default.Compare(x.Weigth,y.Weigth);
        }
    }
}

This code “seems” to work. Is it the correct way ? Does this code prevent threading issues when accessing an existing SortedSet ?

This code does not seems to works. I can see that some of the found values are lost. I suspect the last argument of AddOrUpdate method not to properly lock the inner SortedSet.

Answer 1

Change this part of your code:

(filePath, source) =>
{
    g_Files[fileName].Add(fd);
    return g_Files[fileName];
}

To this:

(filePath,source) =>
{
    source.Add(fd);
    return source;
}