Question

I have a computational algebra task I need to code up. The problem is broken into well-defined individuals tasks that naturally form a tree – the task is combinatorial in nature, so there’s a main task which requires a small number of sub-calculations to get its results. Those sub-calculations have sub-sub-calculations and so on. Each calculation only depends on the calculations below it in the tree (assuming the root node is the top). No data sharing needs to happen between branches. At lower levels the number of subtasks may be extremely large.

I had previously coded this up in a functional fashion, calling the functions as needed and storing everything in RAM. This was a terrible approach, but I was more concerned about the theory then.

I’m planning to rewrite the code in C++ for a variety of reasons. I have a few requirements:

• Checkpointing: The calculation takes a long time, so I need to be able to stop at any point and resume later.
• Separate individual tasks as objects: This helps me keep a good handle of where I am in the computations, and offers a clean way to do checkpointing via serialization.
• Multi-threading: The task is clearly embarrassingly parallel, so it’d be neat to exploit that. I’d probably want to use Boost threads for this.

I would like suggestions on how to actually implement such a system. Ways I’ve thought of doing it:

1. Implement tasks as a simple stack. When you hit a task that needs subcalculations done, it checks if it has all the subcalculations it requires. If not, it creates the subtasks and throws them onto the stack. If it does, then it calculates its result and pops itself from the stack.
2. Store the tasks as a tree and do something like a depth-first visitor pattern. This would create all the tasks at the start and then computation would just traverse the tree.

These don’t seem quite right because of the problems of the lower levels requiring a vast number of subtasks. I could approach it in a iterator fashion at this level, I guess.

I feel like I’m over-thinking it and there’s already a simple, well-established way to do something like this. Is there one?

---

Technical details in case they matter:

• The task tree has 5 levels.
• Branching factor of the tree is really small (say, between 2 and 5) for all levels except the lowest which is on the order of a few million.
• Each individual task would only need to store a result tens of bytes large. I don’t mind using the disk as much as possible, so long as it doesn’t kill performance.
• For debugging, I’d have to be able to recall/recalculate any individual task.
• All the calculations are discrete mathematics: calculations with integers, polynomials, and groups. No floating point at all.

Answer 1

there’s a main task which requires a small number of sub-calculations to get its results. Those sub-calculations have sub-sub-calculations and so on. Each calculation only depends on the calculations below it in the tree (assuming the root node is the top). No data sharing needs to happen between branches. At lower levels the number of subtasks may be extremely large… blah blah resuming, multi-threading, etc.

Correct me if I’m wrong, but it seems to me that you are exactly describing a map-reduce algorithm.

Just read what wikipedia says about map-reduce :

"Map" step: The master node takes the input, partitions it up into smaller sub-problems, and distributes those to worker nodes. A worker node may do this again in turn, leading to a multi-level tree structure. The worker node processes that smaller problem, and passes the answer back to its master node.

"Reduce" step: The master node then takes the answers to all the sub-problems and combines them in some way to get the output – the answer to the problem it was originally trying to solve.

Using an existing mapreduce framework could save you a huge amount of time.

I just google "map reduce C++" and I start to get results, notably one in boost http://www.craighenderson.co.uk/mapreduce/