Question

I have a need to develop a .NET app which is very similar to a web spider/crawler. Get data from a website, process data, save data in a database and send an email.

I want to process as many sites at once as the machine can (within reason). Each process is independent of each other. I will be using some third party server components, like from Chilkat Software. Only a single computer is used. Starting with Windows 7 64bit then going to Windows Server.

What architecture or design should I use which handles the requirements I mentioned? Running several instances of the app (easiest way)? Using Windows WorkFlow Foundation (Never used it)? Some kind of parallel processing? ..?
A pointer to a sample app which follows the proposed design is a plus.

Answer 1

I’d recommend using the System.Threading.Tasks library for something like this.

You could then do something like this in your app:

foreach(var input in listToProcess)
{
  Task.Factory.StartNew(() => ProcessInput(input));
}

private static void ProcessInput(Foo myInput)  // for example, this might be a url in your case
{
  // your specific processing here: get data from site, process, save, email
}