Question

In F# I could use

// Synchronous version
let rec folderCollectorSync path =
    try
        let dirs = Directory.GetDirectories path 
        for z in dirs do folderCollectorSync z
    with
    | ex -> ()

// Asynchronous version that uses synchronous when 'nesting <= 0'
let rec folderCollector path nesting =
    async { if nesting <= 0 then return folderCollectorSync path 
            else 
                try
                    let dirs = Directory.GetDirectories path 
                    do! [for z in dirs -> folderCollector z (nesting - 1) ] 
                        |> Async.Parallel |> Async.Ignore
                with ex -> () }

folderCollector @"C:\" 5 |> Async.RunSynchronously

to travel a directory async for the first 5 levels.

I’ve tried to redo the code above (without the use of Async.Parallel of course).

And it looks something like:

static void TravelSync(string path, CountdownEvent cd)
{
    var dirs = Directory.GetDirectories(path);
    var cdown = new CountdownEvent(dirs.Length);

    foreach (var d in dirs)
        TravelSync(d, cdown);
    cdown.Wait();
    cd.Signal();
}

static void Travel(string path, int nesting, CountdownEvent cd)
{
    if (!Directories.Contains(path))
    {
        if (nesting <= 0)
        {
            TravelSync(path, cd);
        }
        else
        {
            Messages.Add(path);
            Directories.Add(path);

            var dirs = Directory.GetDirectories(path);
            var cdown = new CountdownEvent(dirs.Length);

            foreach (var d in dirs)
                ThreadPool.QueueUserWorkItem(o => Travel(d, nesting - 1, cdown));

            cdown.Wait();
            cd.Signal();
        }
    }
}

By no surprise the c# version is slow as hell, and also it just stops after have crawled 5 directories.

So my question is: How can F# keep track of the async operations? My C# version is poor, and have lots of perfomance problems.

I’m aware of that I simply can use the F# code in my C# project, but since this is just for exercise I’m more interested in how to do it in C#.

Answer 1

Firstly, there is a bug in Travel(), where you queue the thread pool work for each directory. You are capturing d in the lambda, but by the time the lambda runs, d will probably always be the last path in the dirs collection. Here’s the fix for that:

foreach ( var d in dirs )
{
    var d2 = d;
    ThreadPool.QueueUserWorkItem( o => Travel( d2, nesting - 1, cdown ) );
}

Apart from that, you are creating a CountdownEvent for every directory on your disk, which is quite expensive. In fact, the CountdownEvent in TravelSync is redundant as this runs synchronously. You can just get rid of them:

static void TravelSync(string path, CountdownEvent cd)
{
    var dirs = Directory.GetDirectories(path);
    //var cdown = new CountdownEvent(dirs.Length);

    // this is normal synchronous code
    foreach (var d in dirs)
        TravelSync(d, null);

    //cdown.Wait();
    if ( cd != null ) cd.Signal();
}

If you’re using .NET 4.0, you can clean up Travel() as well, using Tasks:

...
else
{
    Messages.Add( path );
    Directories.Add( path );

    try
    {
        var dirs = Directory.GetDirectories( path );

        var tasks = dirs.Select(
            d => Task.Factory.StartNew(
                () => Travel( d, nesting - 1, null )
            )
        ).ToArray();

        Task.WaitAll( tasks );

        foreach ( var t in tasks ) t.Dispose();
    }
    catch ( Exception x )
    {
        ...
    }
}

Of course, the Messages and Directories collections must be thread safe.

---

EDIT: Actually, PLINQ makes this even easier:

    Parallel.ForEach( dirs, d => Travel( d, nesting - 1, null ) );