I’ve got a for loop I want to parallelize with something like PLINQ’s Parallel.ForEach().
The key here is that the C++ library i’m calling to do the computation is decidedly not thread safe, therefore, any plans to parallelize this need to do so across multiple processes.
I was thinking about using WCF to create a “distributor” process to which the “client” and multiple “calculators” could both connect and add/remove items to/from a queue and then the “calculator” sends the results directly back to the client which could update the gui as it receives them. This architecture would allow me to bring as many “calculators” online as I have processors and as I see it even bring them up across multiple computers creating a potential farm of processing power to which all the clients could share.
I’m just wondering if anyone has had any experience doing this and if there are existing application blocks or frameworks that I can use to build this for me. PLINQ does it within the process. is there like a DPLINQ (distributed) or something?
Also if that doesn’t exist, does anybody want to give an opinion on my proposed architecture? Any obvious pitfalls? Does anyone think it will work!?!?!?
Sounds like you could be looking for Dryad. It’s a Microsoft research project right now, but they do have an “academic release” available. My understanding is that they are also in the process of better productizing it (probably some kind of integration with Azure) for RTM sometime near the end of 2011. Mary Jo Foley covers more about this here.