none
CPU time spent in MPI_Barrier()

    Question

  • This entry has been transferred from the archived forum site and was originally posted on 3/14/2008 by User ID "Flame"

     

    Hello,

    I used microsoft mpi (msmpi) to do message passing for a CFD fortran code. In the code, there are several barriers to synchronize the code. I measured the wall clock time and cpu time spent for calling MPI_Barrier(). I would expect that the cpu time to call MPI_Barrier() is approximately zero and the wall clock time depends on the idle time. However, with msmpi, I observed that the CPU time and the wall clock time to call MPI_Barrier() are always approximately equal. It seems that when a processor is idle the CPU time on the processor still accumulates. Does anyone know what's going on? Do I need to specify some flags to start the mpi job with mpiexec? The test code is in the following:

    Thanks,
    Flame
    -------------------------------------------------------------------
    program master

    use mpi
    implicit none
    integer :: C1,C2,C3,R,M
    real :: tm0,tm1,tm2
    integer :: i,ierr,np,myid,isum,ip,istatus(MPI_STATUS_SIZE)

    call SYSTEM_CLOCK (COUNT=C1,COUNT_RATE=R,COUNT_MAX=M)
    call CPU_TIME(tm0)

    call MPI_init (ierr)
    call MPI_comm_size (MPI_COMM_WORLD, np, ierr)
    call MPI_comm_rank (MPI_COMM_WORLD, myid, ierr)

    if(myid==0) then
    ! do some job only on processor 0
    do i=1,200000000
    C2 = 0
    C2 = sin(0.123)
    C2 = cos(0.0134)
    C2 = exp(dble(C2))
    C2 = log(dble(C2))
    enddo
    endif

    call SYSTEM_CLOCK (COUNT=C2,COUNT_RATE=R,COUNT_MAX=M)
    call CPU_TIME(tm1)

    write(*,*)
    'Computation+++++++++++++++++++++++++++++++++++++++++++++++++++++++'
    write(*,*) 'Wall Time=', dble(C2-C1)/dble(R)
    write(*,*) 'CPU Time=', tm1-tm0

    call SYSTEM_CLOCK (COUNT=C2,COUNT_RATE=R,COUNT_MAX=M)
    call CPU_TIME(tm1)

    call MPI_barrier(MPI_COMM_WORLD,ierr)

     call SYSTEM_CLOCK (COUNT=C3,COUNT_RATE=R,COUNT_MAX=M)
    call CPU_TIME(tm2)
    !tm2 = DCLOCK()

    write(*,*)
    'Barrier+++++++++++++++++++++++++++++++++++++++++++++++++++++++'
    write(*,*) 'Wall Time=', dble(C3-C2)/dble(R)
    write(*,*) 'CPU Time=', tm2-tm1

    write(*,*) 'Total+++++++++++++++++++++++++++++++++++++++++++++++++++++++'
    write(*,*) 'Wall Time=', dble(C3-C1)/dble(R)
    write(*,*) 'CPU Time=', tm2-tm0

    call MPI_finalize (ierr)

    end program master
    -------------------------------------------------------------------
    Monday, March 24, 2008 5:05 PM

Answers

  • Hi Michael,

     

    MSMPI v1 is in polling for the communcation, so while its busy waiting the clock time and the cpu time would be the same. you can disable the polling in v1 by disabeling shared memory; add "-env MPICH_DISABLE_SHM 1" to your mpiexec command line.

     

    MSMPI v2 internals improved on that and it knowns to block and still provide better performance.

     

    hope this helps,

    .Erez

     

    Friday, March 28, 2008 4:29 AM