Question

I’m writing a C# console app and having an issue where the primary thread closes before all the spawned off Tasks have a chance to complete. General program flow looks like this:

The program takes the path to a directory as a command line parameter. It then iterates over every file in that directory matching a specific extension and spawns a new task for each matching file to open up and analyze that file:

Directory.GetFiles(directoryPath).ToList<string>().Where(x => Path.GetExtension(x) == ".xfast").ToList().ForEach(x => ProcessXFastFile(x));

Each of these files contains a large number of CSV values corresponding back to a unique product key. For example, one file might contain 1 million CSV lines and 400 unique product keys. The point of the program is to sort all of the lines based on the product key and write them back out to individual files based on the key.

So in the ProcessXFastFile method I create a Dictionary<string, List<string>> using the product codes as Key values and adding each CSV line that matches that key into the corresponding list.

Once the sorting is finished, I iterate over the dictionary and spawn off a new task for each Key value and write out the CSV lines in that List:

foreach (KeyValuePair<string, List<string>> entry in productDictionary) {
    string writePath = String.Format(@"{0}\{1}-{2}.txt", directoryPath, hour, entry.Key);
    List<string> quotes = entry.Value;
    Task writeFileTask = Task.Factory.StartNew(() => WriteProductFile(writePath, quotes));
}

The problem is that the primary thread is ending before each of these Tasks to WriteProductFile have a chance to finish writing out their data. For example, I use one file as a control test and know that at there are ~450 unique keys in the dictionary so there should be that many corresponding files written out. However, only about 10 files get a chance to write out before the program ends.

How can I ensure that my program stays live long enough for all Tasks to finish? Any help is appreciated.

Answer 1

You can put all of the tasks you get into a List<Task> instead of just creating them and tossing them, and then use the Task.WaitAll() to wait for completion:

var tasks = new List<Task>();

foreach (KeyValuePair<string, List<string>> entry in productDictionary) {
    string writePath = String.Format(@"{0}\{1}-{2}.txt", directoryPath, hour, entry.Key);
    List<string> quotes = entry.Value;

    // add task to list
    tasks.Add(Task.Factory.StartNew(() => WriteProductFile(writePath, quotes)));
}

// now wait for all tasks to finish, you'd also want to handle  exceptions of course.
Task.WaitAll(tasks.ToArray());

There are many variations of WaitAll(), you could wait indefinitely (as above), or wait for a TimeSpan and if time-out then print a progress message and wait again, etc…