Question

I have a consumer that enumerates on a BlockingCollection<T> where T : IComparable then writes a row to a text file for each item. The end result are text files of a maximum of about 7Gb-10Gb. However they are not in order.

I cannot sort the list in memory as it could end up being very big. The only way I could think of toward a solution is to write to temp text files in order they arrive then shuffling them at the end but I can’t imagine that’s very efficient.

Any ideas on how this can be done?

Answer 1

I ended up loading the results into a staging table in SQL using SSIS and sorted them that way.