Question

I will say this right off the bat. I am an amateur at threading. I am a senior c# web developer, but I have a project that requires me to populate a lot of objects that take a long time to populate as they require WebRequests and Responses to populate. I have everything working without threading, but it does not run fast enough for my requirements. I would like to pass everything to a ThreadPool to have the threading managed for me as I may be queuing up 20,000 threads at the same time and for obvious reasons. I do not want to hit a website with the requests needed to populate all of them at once.

What I would like to do is to pass in an object, populate it, and then add it to a collection in the main thread once it is populated. Then once all the objects are populated, continue on with execution of the program. I do not know how many objects will need to be populated until they are all populated either.

My question…What is the best approach to doing this?

Here is the loop that I am trying to speed up:

foreach (HElement hElement in repeatingTag.RunRepeatingTagInstruction())
{
    object newObject = Activator.CreateInstance(currentObject.GetType().GetGenericArguments()[0]);
    List<XElement> ordering = GetOrdering(tagInstructions.Attribute("type").Value);
    RunOrdering(ordering, newObject, hElement);
    MethodInfo method = currentObject.GetType().GetMethod("Add");
    method.Invoke(currentObject, new[] { newObject });
}

I don’t know what the object is beforehand so I create it using the Activator. The RunOrdering method runs through the instructions that I pass that tell it how to populate the object. Then I add it to the collection. Also, the object itself may have properties that will require this method to run through and populate their data.

Answer 1

Since you probably have to wait for them all to be complete, all you need is a Parallel.ForEach() or equivalent. And a Thread-safe collection. Note that for I/O intensive tasks you would want to limit the number of Threads. 20.00 threads would be insane in any situation.

But we would need to see more details (code). Note that there is no such thing as “a collection in the main thread”.