Question

In a cluster running MPI code, is a copy of all the declared-variables, sent to all nodes , so that all nodes can access it locally, and not perform a remote memory access ?

Answer 1

No, MPI itself can’t do this for you in single call.

There is an own state of memory in every MPI process, and every value may be different in any of MPI process.

The only way of sending/receiving data is to use explicit calls of MPI, like Send or Recv. You can pack most of your data into some memory space and send this area of memory to each MPI Process, but this area will not contain ‘every declared variable’, only variables placed manually into this area.

Update:

Each node runs a copy of the program. Each copy will initialize variables as it want (it can be the same initialization, or individual, based on MPI Process number, called Rank; got from MPI_Comm_Rank function). So every variable exist in N copyes; one set per MPI Process. Every process sees variables, but only the set it owns. Values of variables are unsyncronized automatically.

So, task of programmer is to syncronize values of variables between Nodes (mpi processes).
E.g. here is small MPI program to compute Pi:

http://www.mcs.anl.gov/research/projects/mpi/usingmpi/examples/simplempi/cpi_c.htm

It will send value of the ‘n‘ variable from first process to all other (MPI_Bcast); and every process will send its own ‘mypi‘ after calculation into ‘pi‘ variable of first process (with addition of individual values via MPI_Reduce function).

Only first process will be able to read N from user (via scanf) and this code is conditionally executed based on rank of process; other processes must get the N from the first because they didn’t read it from user directly.

Update2 (sorry for late answer):
This is syntax of MPI_Bcast. Programmer should give an address of variable into this function. Each of MPI processes will give the address of its own ‘n’ variable (it can be different). And the MPI_Bcast will

• check the rank of current process and compare with other argument, the rank of “Broadcaster”.
• If the current process is broadcaster, MPI_Bcast will read value, placed in memory at given address (it will read value of the ‘n’ variable on “Broadcaster”); then the value will be send via network.
• Else, if the current process is not a broadcaster, it is an “receiver”. MPI_Bcast at receiver will get the value from “Broadcaster” (Using MPI Library internals, via network) and store the value in memory of current process at given address.

So, the address is given to this function because on some nodes the function will write to the variable. Only value is send via network.