Question

When I try to send MPI derived datatypes with “large” arrays (2 arrays of 100 000 floats each), my program segfaults. It runs normally with smaller arrays though.

Below is a small reproducible example.
This small program segfaults with the following MPI implementation : IntelMPI, BullXMPI.
It works normally with OpenMPI and PlatformMPI.
Here is a log with a backtrace of the example : http://pastebin.com/FMBpCuj2

Changing mpi_send to mpi_ssend does not help. However, mpi_send with a single larger array of 2*100 000 floats works fine. In my opinion, this points to an issue with derived datatypes.

program struct 
include 'mpif.h' 

type Data
  integer :: id
  real, allocatable :: ratio(:)
  real, allocatable :: winds(:)
end type 

type (Data) :: test
integer :: datatype, oldtypes(3), blockcounts(3) 
integer :: offsets(3)
integer :: numtasks, rank, i,  ierr 
integer :: n, status(mpi_status_size)

call mpi_init(ierr) 
call mpi_comm_rank(mpi_comm_world, rank, ierr) 
call mpi_comm_size(mpi_comm_world, numtasks, ierr) 

if (numtasks /= 2) then
  write (*,*) "Needs 2 procs"
  call exit(1)
endif

n = 100000
allocate(test%ratio(n))
allocate(test%winds(n))
if (rank == 0) then
  test%ratio = 6
  test%winds = 7
  test%id = 2
else
  test%id = 0
  test%ratio = 0
  test%winds = 0
endif

call mpi_get_address(test%id, offsets(1), ierr)
call mpi_get_address(test%ratio, offsets(2), ierr)
call mpi_get_address(test%winds, offsets(3), ierr)

do i = 2, size(offsets)
  offsets(i) = offsets(i) - offsets(1)
end do
offsets(1) = 0

oldtypes = (/mpi_integer, mpi_real, mpi_real/)
blockcounts = (/1, n, n/)

call mpi_type_struct(3, blockcounts, offsets, oldtypes, datatype, ierr) 
call mpi_type_commit(datatype, ierr) 

if (rank == 0) then 
  !call mpi_ssend(test, 1, datatype, 1, 0,  mpi_comm_world, ierr) 
  call mpi_send(test, 1, datatype, 1, 0,  mpi_comm_world, ierr) 
else
  call mpi_recv(test, 1, datatype, 0, 0,  mpi_comm_world, status, ierr) 
end if

print *, 'rank= ',rank
print *, 'data= ',test%ratio(1:5),test%winds(1:5)

deallocate (test%ratio)
deallocate (test%winds)
call mpi_finalize(ierr) 


end 

Note : The comparison between different MPI implentations is not objective as the tests were not all on the same machine (some of them are supercomputers). Still, I don’t think it should make a difference.

Edit : The code works with static arrays. And this is Fortran 90.

Answer 1

Can I suggest you use a debugger? I just tried your example in Allinea DDT and saw the problem in two minutes flat. You need to use a debugger – your code "looks right", so it’s time to watch how it behaves in practice.

I clicked to turn on memory debugging (a way to force some hidden errors to show) and your example crashed with OpenMPI then every time. The crash was in the sender.

So, I started to step through with DDT – with DDT’s memory debugging turned on.

First, you call MPI_Get_address – to fill an array of offsets. Take a look at those offsets! The integer’s address is positive, and the allocatable arrays offsets are negative: a bad sign. the addresses have overflowed.

The addresses of the allocated data will be in a very different area of memory to the statically allocated integer. All bets are off if you use 32-bit arithmetic to manipulate a 64 bit pointer (MPI_Get_address warns about this). With static arrays it didn’t crash because their addresses would be close enough to that of the integer to not overflow.

You send this incorrect offsets array to MPI_Send, it reads the data from where it should not (look at the offset buffer again to convince yourself) and hence segfaults.

The real fix here is —

1. With MPI_Get_address – use INTEGER(KIND=MPI_ADDRESS_KIND) for the declaration of offset – to ensure a 64-bit code gets 64-bit integers.

2. The MPI_type_struct should be replaced with MPI_type_create_struct – the former is deprecated and does not take the offsets in the form of MPI_ADDRESS_KIND integers, only 4-byte integers – and hence is flawed.

With those changes, your code runs.

Good luck!