Answered by:
Using MPI_IN_PLACE

Question
-
I am using MPI_IN_PLACE for MPI_ALLREDUCE operation. My OS is Win Server 2008 HPC Edition and HPC PACK 2008 SDK SP2.
When I use:
CALL MPI_ALLREDUCE(MPI_IN_PLACE,in,1,
& MPI_DOUBLE_PRECISION, MPI_SUM,mpi_comm_world,ierr)I get incorrect results in variable "in". If I use the regular operation:
CALL MPI_ALLREDUCE(in,out,1,
& MPI_DOUBLE_PRECISION, MPI_SUM,mpi_comm_world,ierr)I get correct results in variable "out".
The code is written in FORTRAN.
Any solutions to the MPI_IN_PLACE issue?
Thanks,
GK.
Wednesday, July 28, 2010 9:34 PM
Answers
-
Thanks GK for the try!
The problem is figurered out to be the option /Qlowercase. Please recompile your program with /Qlowercase option and try again. Please let me know whehter it works.
However, I don't know why this option can have effect on MPI_IN_PLACE. You may need direct this question to Intel compiler team.
Thanks,
James
- Marked as answer by GK-F3D Monday, September 20, 2010 2:51 PM
Friday, September 17, 2010 9:11 PM
All replies
-
As I know, there is no known issue for MPI_IN_PALCE for MPI_Allreduce of MSMPI. I did a quick check with C, there is no problem to run with MPI_IN_PLACE.
Could you double check the values in "in" and "out"? Also, could you post your Fortran source code here so I can take a try?
Thanks,
James
Wednesday, August 4, 2010 9:07 PM -
Hi James,
I have checked the "in" and "out" values and I'm still having the same issue. This is prevalent in other MPI calls and is not limited to MPI_Allreduce. Here is the simple code I used for your reference;
FORTRAN 90 Code:
program allreduce_check
implicit none
include "C:\Program Files\Microsoft HPC Pack 2008 SDK\
1Include\mpif.h"
integer, parameter :: n=12
integer :: ierr,iproc,nproc,imin,n_,imin_,imax_,q
integer :: i
double precision :: pmmfpk
double precision :: pmmfpk_mpi_in,pmmfpk_mpi_out
call mpi_init(ierr)
call mpi_comm_rank(mpi_comm_world, iproc, ierr)
call mpi_comm_size(mpi_comm_world, nproc, ierr)
c Works only for even number of nodes and proc
q=n/nproc
imin=1
n_=q
imin_=imin+iproc*q
imax_=imin_+n_-1
pmmfpk = (imin_+imax_)/2.d0 + iproc*3.2d0
c print*, 'rank=',iproc,'pmmfpk=',pmmfpk
pmmfpk_mpi_in = pmmfpk
c CALL MPI_ALLREDUCE(pmmfpk_mpi_in,pmmfpk_mpi_out,1,
c & MPI_DOUBLE_PRECISION, MPI_SUM,mpi_comm_world,ierr)
CALL MPI_ALLREDUCE(MPI_IN_PLACE,pmmfpk_mpi_in,1,
& MPI_DOUBLE_PRECISION, MPI_SUM,mpi_comm_world,ierr)
print*, 'rank=',iproc,'pmmfpk_mpi',pmmfpk_mpi_in
call mpi_finalize(ierr)
end program allreduce_check
I'm eager to get some solution for this problem and thanks in advance.
-GK
Thursday, August 19, 2010 10:43 PM -
Thanks for the source code.
I tried with 2 4 and 32 ranks, verified that both ways returned the same results. The results for 2 processes are 16.2, 4 are 45.2 and 32 are 1603.2.
I was wondering what results you got?
Thanks,
James
Friday, August 20, 2010 7:58 PM -
Hi James,
Thanks for the prompt reply. I'm using the following command to compile;
ifort allreduce.F "C:\Program Files\Microsoft HPC Pack 2008 SDK\Lib\amd64\msmpi.lib" "C:\Program Files\Microsoft HPC Pack 2008 SDK\Lib\amd64\msmpifec.lib"
And the following results are from the runs;
1) without MPI_IN_PLACE (using "in" and "out")
c:\Users\Administrator>mpiexec -np 2 allreduce.exe rank= 0 pmmfpk_mpi_in= 3.50000000000000 rank= 1 pmmfpk_mpi_in= 12.7000000000000 rank= 1 pmmfpk_mpi_out 16.2000000000000 rank= 0 pmmfpk_mpi_out 16.2000000000000
2) with MPI_IN_PLACE
c:\Users\Administrator>mpiexec -np 2 allreduce.exe rank= 0 pmmfpk_mpi_in= 3.50000000000000 rank= 1 pmmfpk_mpi_in= 12.7000000000000 rank= 0 pmmfpk_mpi_out 0.000000000000000E+000 rank= 1 pmmfpk_mpi_out 0.000000000000000E+000
Also I would like you to know that I'm using;
1) Windows Server 2008 HPC Edition.
2) Microsoft HPC Pack 2008 SDK (Service Pack 2).
Thanks,
GK
Friday, August 20, 2010 9:06 PM -
Hello GK,
FOr MPI_IN_PLACE, it means that the input buffer will be replaced with the output data. In your output with MPI_IN_PLACE, the pmmfpk_mpi_out will be zeros as expected. Could you check the value of pmmfpk_mpi_in after calling MPI_ALLREDUCE with MPI_IN_PLACE? It should be the value of 16.2.
Thanks,
James
Thursday, August 26, 2010 3:35 PM -
Hi James,
I'm printing the pmmfpk_mpi_in but I have just named it pmmfpk_mpi_out. I'm checking the input buffer only and not the output data.
-GK
Tuesday, September 14, 2010 8:02 PM -
I've tried with Microsoft HPC Pack 2008 SDK (Service Pack 2) on Windows Server 2008 HPC Edition, still there is no repro of your program.
Could you please try compile and execute the following simplified code with >=2 ranks and post your result here?
program main implicit none include 'mpif.h' integer pmmfpk_mpi_in, ierr, iproc call mpi_init(ierr) call mpi_comm_rank(mpi_comm_world, iproc, ierr) pmmfpk_mpi_in = 1 CALL MPI_ALLREDUCE(MPI_IN_PLACE,pmmfpk_mpi_in,1, & MPI_INTEGER, MPI_SUM,mpi_comm_world,ierr) print*, 'rank=',iproc,'pmmfpk_mpi_in = ',pmmfpk_mpi_in call mpi_finalize(ierr) end program
Thanks,
James
Friday, September 17, 2010 3:57 PM -
Hi James,
I compiled the code u sent in the following way;
c:\Users\Administrator>ifort james.F "C:\Program Files\Microsoft HPC Pack 2008 SDK\Lib\amd64\msmpi.lib" "C:\Program Files\Microsoft HPC Pack 2008 SDK\Lib\amd64\msmpifec.lib"
Intel(R) Visual Fortran Intel(R) 64 Compiler Professional for applications running on Intel(R) 64, Version 11.1 Build 20090930 Package ID: w_cprof_p_11.1.048
Copyright (C) 1985-2009 Intel Corporation. All rights reserved.
Microsoft (R) Incremental Linker Version 8.00.50727.762
Copyright (C) Microsoft Corporation. All rights reserved.
-out:james.exe
-subsystem:console
james.obj
"C:\Program Files\Microsoft HPC Pack 2008 SDK\Lib\amd64\msmpi.lib"
"C:\Program Files\Microsoft HPC Pack 2008 SDK\Lib\amd64\msmpifec.lib"
AND THIS IS the output from the program;
c:\Users\Administrator>mpiexec -np 4 james.exe
rank= 2 pmmfpk_mpi_in = 0
rank= 3 pmmfpk_mpi_in = 0
rank= 1 pmmfpk_mpi_in = 0
rank= 0 pmmfpk_mpi_in = 0Thanks,
GK
Friday, September 17, 2010 4:29 PM -
Thanks GK for the try!
The problem is figurered out to be the option /Qlowercase. Please recompile your program with /Qlowercase option and try again. Please let me know whehter it works.
However, I don't know why this option can have effect on MPI_IN_PLACE. You may need direct this question to Intel compiler team.
Thanks,
James
- Marked as answer by GK-F3D Monday, September 20, 2010 2:51 PM
Friday, September 17, 2010 9:11 PM -
Thanks James,
I tried the /Qlowercase option and it worked. I feel this has to do with MSMPI and not the compiler. I suspect this because only the MPI_IN_PLACE is affected. It also works well on Linux.
I'll forward this to Intel as well and find out what they think. Thanks again for your help.
-GK
Monday, September 20, 2010 3:01 PM -
Hi GK,
The problem looks not related with Intel compiler. The root cause is that one Fortran program (for c-bindings) is compiled with /Qlowercase (msmpifec.lib). So the linker only recognized the lower case symbols.
We are working on a generic solution for this problem. And for now, you need use /Qlowercase option as the workaround.
Thanks,
James
Monday, September 20, 2010 8:40 PM