Home/ Questions/Q 6707711
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-26T07:40:19+00:00

We have developed an algorithm library in C++ which allows the user to implement

  • 0

We have developed an algorithm library in C++ which allows the user to implement his own datatypes for sharing data between individual algorithms (also implemented by the user).
This works fine, but we want to provide parallelization at library level. The individual algorithms should be executed in parallel on different nodes of distributed memory machines.

We decided to use MPI for parallelization, as it can be used for distributed and shared memory machines without code changes.
Unfortunately we fight now the problem how to distribute the user implemented datatypes between the nodes. We have the following problems:

  • We do not know how big the data might be, it might even change from run to run.
  • We do not know what data is inside the data structure.
  • The amount of data can be very big up to 1GB (this should be no problem with MPI)
  • The user should not see any difference in implementing the datatypes or algorithms for parallel execution (for the algorithm there is actually no problem)

Is there a possibility to use MPI to share these data between the nodes, or are there approaches available, which might be better suited for this kind of problem.
We would like to have a solution which works at least on shared memory machines however we would love to have a solution which works without code changes on shared and distributed memory machines.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    Yes, you can do this with MPI, but no, MPI can’t do it for you by itself.

    Whether you’re sending this data to another node, or writing it to disk, at some point you need to expressly describe the data structures layout in memory so that it can be serialized. If you pass MPI (or any other communications library) a pointer, it doesn’t know what lies on the other side of that pointer, and so it has no way of traversing the data structure to copy its contents.

    You can marshal the arguments into plain old data (manually, or with things like MPI_PACK), or you can create an MPI datatype which describes the layout of data in memory for that particular instance, and that will copy the data over. In addition, you’ll need to redirecting any pointers within the data structure. Boost serialization may be able to help you with all of this.

    • 0