Question

I’m using a parallel for loop in my code to run a long running process on a large number of entities (12,000).

The process parses a string, goes through a number of input files (I’ve read that given the number of IO based things the benefits of threading could be questionable, but it seems to have sped things up elsewhere) and outputs a matched result.

Initially, the process goes quite quickly – however it ends up slowing to a crawl. It’s possible that it’s just hit a number of particularly tricky input data, but this seems unlikely looking closer at things.

Within the loop, I added some debug code that prints “Started Processing: ” and “Finished Processing: ” when it begins/ends an iteration and then wrote a program that pairs a start and a finish, initially in order to find which ID was causing a crash.

However, looking at the number of unmatched ID’s, it looks like the program is processing in excess of 400 different entities at once. This seems like, with the large number of IO, it could be the source of the issue.

So my question(s) is(are) this(these):

• Am I interpreting the unmatched ID’s properly, or is there some clever stuff going behind the scenes I’m missing, or even something obvious?
• If you’d agree what I’ve spotted is correct, how can I limit the number it spins off and does at once?

I realise this is perhaps a somewhat unorthodox question and may be tricky to answer given there is no code, but any help is appreciated and if there’s any more info you’d like, let me know in the comments.

Answer 1

Without seeing some code, I can guess at the answers to your questions:

• Unmatched IDs indicate to me that the thread that is processing that data is being de-prioritized. This could be due to IO or the thread pool trying to optimize, however it seems like if you are strongly IO bound then that is most likely your issue.
• I would take a look at Parallel.For, specifically using ParallelOptions.MaxDegreesOfParallelism to limit the maximum number of tasks to a reasonable number. I would suggest trial and error to determine the optimum number of degrees, starting around the number of processor cores you have.

Good luck!