Question

I’ve read a lot recently about how writing multi-threaded apps is a huge pain in the neck, and have learned enough about the topic to understand, at least at some level, why it is so.

I’ve read that using functional programming techniques can help alleviate some of this pain, but I’ve never seen a simple example of functional code that is concurrent. So, what are some alternatives to using threads? At least, what are some ways to abstract them away so you needn’t think about things like locking and whether a particular library’s objects are thread-safe.

I know Google’s MapReduce is supposed to help with the problem, but I haven’t seen a succinct explanation of it.

Although I’m giving a specific example below, I’m more curious of general techniques than solving this specific problem (using the example to help illustrate other techniques would be helpful though).

I came to the question when I wrote a simple web crawler as a learning exercise. It works pretty well, but it is slow. Most of the bottleneck comes from downloading pages. It is currently single threaded, and thus only downloads a single page at a time. Thus, if the pages can be downloaded concurrently, it would speed things up dramatically, even if the crawler ran on a single processor machine. I looked into using threads to solve the issue, but they scare me. Any suggestions on how to add concurrency to this type of problem without unleashing a terrible threading nightmare?

Answer 1

I’ll add an example of how functional code can be used to safely make code concurrent.

Here is some code you might want to do in parallel, so you don’t have wait for one file to finish to start downloading the next:

void DownloadHTMLFiles(List<string> urls) {     foreach(string url in urls)     {          DownlaodOneFile(url);  //download html and save it to a file with a name based on the url - perhaps used for caching.     } } 

If you have a number of files the user might spend a minute or more waiting for them all. We can re-write this code functionally like this, and it basically does the exact same thing:

urls.ForEach(DownloadOneFile); 

Note that this still runs sequentially. However, not only is it shorter, we’ve gained an important advantage here. Since each call to the DownloadOneFile function is completely isolated from the others (for our purposes, available bandwidth isn’t an issue) you could very easily swap out the ForEach function for another very similar function: one that kicks off each call to DownlaodOneFile on a separate thread from a threadpool.

It turns out .Net has just such a function availabe using Parallel Extensions. So, by using functional programming you can change one line of code and suddenly have something run in parallel that used to run sequentially. That’s pretty powerful.