none
Get MS-MPI app to run over NetworkDirect RRS feed

  • Question

  • Hello,

    I have setup a RoCEv2 connection between two Windows 10 Pro PCs with Mellanox Connect-X 4 VPI 56Gps cards using WinOF-2 drivers. I can get nd_send_lat to work between the two:

    C:\Windows\system32>start /b /affinity 0x1 nd_send_lat -a -C 192.168.2.2
    
     #bytes #iterations   Avg.[usec]     Gb/s     CPU Util.
     1         100000       1.53         0.01     100.00
     2         100000       1.02         0.02     100.00
     4         100000       1.53         0.02     100.00
     8         100000       1.53         0.04     100.00
     16        100000       1.53         0.08     100.00
     32        100000       1.53         0.17     100.00
     64        100000       1.53         0.33     100.00
     128       100000       2.05         0.50     100.00
     256       100000       2.05         1.00     100.00
     512       100000       2.56         1.60     100.00
     1024      100000       2.56         3.20     100.00
     2048      100000       3.07         5.34     100.00
     4096      100000       3.07         10.67    100.00
     8192      100000       4.09         16.00    100.00
     16384     100000       5.12         25.61    100.00
     32768     100000       7.68         34.14    100.00
     65536     100000       13.31        39.39    100.00
     131072    100000       23.55        44.52    100.00
     262144    100000       44.54        47.08    100.00
     524288    100000       86.02        48.76    100.00
     1048576   100000       168.96       49.65    100.00
     2097152   100000       334.85       50.10    100.00
     4194304   100000       667.13       50.30    100.00

    I suppose that would mean the hardware part is properly configured and working. However, trying to pass an mpi ping-pong test over NetworkDirect fails:

    C:>mpiexec.exe /hosts 2 192.168.2.1 192.168.2.2 /env MSMPI_ND_ZCOPY_THRESHOLD -1 /env MSMPI_DISABLE_ND 0 /env MSMPI_DISABLE_SOCK 1 /affinity /priority 1 mpipingpong -r -op -pc
    
    [ DESKTOP-MG0GUD5#1 ] Fatal Error: MPI Failure
    [ DESKTOP-MG0GUD5#1 ] Error Details:
    [ DESKTOP-MG0GUD5#1 ]   Other MPI error, error stack:
    [ DESKTOP-MG0GUD5#1 ]   MPI_Allgather(sbuf=0x0000000000E7F940, scount=128, MPI_CHAR, rbuf=0x0000000003561EC0, rcount=128, MPI_CHAR, MPI_COMM_WORLD) failed
    [ DESKTOP-MG0GUD5#1 ]   unable to connect to 192.168.2.1 192.168.0.15 DESKTOP-C61LVC8  on port 0, the socket interconnect is disabled
    
    
    [ DESKTOP-C61LVC8#0 ] Fatal Error: MPI Failure
    [ DESKTOP-C61LVC8#0 ] Error Details:
    [ DESKTOP-C61LVC8#0 ]   Other MPI error, error stack:
    [ DESKTOP-C61LVC8#0 ]   MPI_Allgather(sbuf=0x0000000000D8FD50, scount=128, MPI_CHAR, rbuf=0x0000000000DE1B00, rcount=128, MPI_CHAR, MPI_COMM_WORLD) failed
    [ DESKTOP-C61LVC8#0 ]   unable to connect to 192.168.2.2 192.168.0.24 DESKTOP-MG0GUD5  on port 0, the socket interconnect is disabled
    
    
    job aborted:
    [ranks] message
    
    [0] application aborted
    aborting MPI_COMM_WORLD (comm=0x44000000), error 250, comm rank 0
    
    [1] application aborted
    aborting MPI_COMM_WORLD (comm=0x44000000), error 250, comm rank 1
    
    ---- error analysis -----
    
    [0] on 192.168.2.1
    mpipingpong aborted the job. abort code 250
    
    [1] on 192.168.2.2
    mpipingpong aborted the job. abort code 250
    
    ---- error analysis -----

    Omitting /env MSMPI_DISABLE_SOCK 1 runs the test over TCP/IP, see my previous post for background information:

    Typical MPI ping-pong latency over RoCE with NetworkDirect

    I am out of ideas and would greatly appreciate any suggestions.

    Best Regards,

    Costas

    Thursday, April 19, 2018 6:20 AM

All replies

  • Hello,

    I did not mention that I am using the MS-MPI Launch service. Using SMPD in debug mode, I managed to get Network Direct working. Now the question becomes:

    Does MS-MPI Launch service support NetworkDirect?

    Costas

    Monday, April 23, 2018 5:35 AM
  • Hi Costas,

    Yes, MSMPILaunchSvc supports NetworkDirect.

    From the logs, it looks odd that the client is trying port 0.
    What is the port that MSMPILaunchSvc is started with? Can you try it in a different port and specify that port in mpiexec command?

    eg: sc start MSMPILaunchSvc -p <portX>
    mpiexec -p <portX> ...

    Regards,
    Jithin.

    Monday, April 23, 2018 6:59 PM
  • Hello Jithin,

    I ve never specified a port when starting msmpilaunchsvc, yet always worked in tcp/ip. I started in port 5555 in both clients and run the ping pong test again:

    C:\mpiexec.exe -d 3 -p 5555 /hosts 2 192.168.2.1 192.168.2.2 /env MSMPI_ND_PORT_RANGE 5555,5555 /env MPICH_NETMASK 192.168.2.0/255.255.255.0 /env MSMPI_DISABLE_ND 0 /env MSMPI_DISABLE_SOCK 1 /affinity /priority 1 mpipingpong -r -op -pc
    [00:8364] host tree:
    [00:8364]  host: 192.168.2.1, parent: 0, id: 1
    [00:8364]  host: 192.168.2.2, parent: 1, id: 2
    [00:8364] mpiexec started smpd manager listening on port 50810
    [00:8364] using spn RestrictedKrbHost/192.168.2.1 to contact server
    [00:8364] DESKTOP-C61LVC8 posting a re-connect to 192.168.2.1:50815 in left child context.
    [00:8364] Authentication completed. Successfully obtained Context for Client.
    [00:8364] Authorization completed.
    [00:8364] version check complete, using PMP version 4.
    [00:8364] creating connect command for left node
    [00:8364] creating connect command to '192.168.2.2'
    [00:8364] posting command SMPD_CONNECT to left child, src=0, dest=1.
    [00:8364] host 192.168.2.2 is not connected yet
    [00:8364] Handling cmd=SMPD_CONNECT result
    [00:8364] cmd=SMPD_CONNECT result will be handled locally
    [00:8364] successful connect to 192.168.2.2.
    [00:8364] posting command SMPD_COLLECT to left child, src=0, dest=1.
    [00:8364] posting command SMPD_COLLECT to left child, src=0, dest=2.
    [00:8364] Handling cmd=SMPD_COLLECT result
    [00:8364] cmd=SMPD_COLLECT result will be handled locally
    [00:8364] Handling cmd=SMPD_COLLECT result
    [00:8364] cmd=SMPD_COLLECT result will be handled locally
    [00:8364] Finished collecting hardware summary.
    [00:8364] posting command SMPD_STARTDBS to left child, src=0, dest=1.
    [00:8364] Handling cmd=SMPD_STARTDBS result
    [00:8364] cmd=SMPD_STARTDBS result will be handled locally
    [00:8364] start_dbs succeeded, kvs_name: '7a914c7e-a30e-48af-b6ae-0ac3db8a385f', domain_name: '4fc8ab52-8f25-4539-baea-34377a11e5fc'
    [00:8364] creating a process group of size 2 on node 0 called 7a914c7e-a30e-48af-b6ae-0ac3db8a385f
    [00:8364] launching the processes.
    [00:8364] posting command SMPD_LAUNCH to left child, src=0, dest=1.
    [00:8364] posting command SMPD_LAUNCH to left child, src=0, dest=2.
    [00:8364] Handling cmd=SMPD_LAUNCH result
    [00:8364] cmd=SMPD_LAUNCH result will be handled locally
    [00:8364] successfully launched process 0
    [00:8364] root process launched, starting stdin redirection.
    [00:8364] Handling cmd=SMPD_LAUNCH result
    [00:8364] cmd=SMPD_LAUNCH result will be handled locally
    [00:8364] successfully launched process 1
    [00:8364] Authentication completed. Successfully obtained Context for Client.
    [00:8364] Authorization completed.
    [00:8364] handling command SMPD_INIT src=1 ctx_key=0
    [00:8364] init: 0:2:7a914c7e-a30e-48af-b6ae-0ac3db8a385f
    [00:8364] handling command SMPD_INIT src=2 ctx_key=0
    [00:8364] init: 1:2:7a914c7e-a30e-48af-b6ae-0ac3db8a385f
    [00:8364] handling command SMPD_STDOUT src=2
    [00:8364] Handling SMPD_STDOUT
    
    [ DESKTOP-MG0GUD5#1 ] Fatal Error: MPI Failure
    [ DESKTOP-MG0GUD5#1 ] Error Details:
    [ DESKTOP-MG0GUD5#1 ]   Other MPI error, error stack:
    [ DESKTOP-MG0GUD5#1 ]   MPI_Allgather(sbuf=0x00000000004EF770, scount=128, MPI_CHAR, rbuf=0x0000000002D91B00, rcount=128, MPI_CHAR, MPI_COMM_WORLD) failed
    [ DESKTOP-MG0GUD5#1 ]   unable to connect to 192.168.2.1 192.168.0.15 DESKTOP-C61LVC8  on port 0, the socket interconnect is disabled
    
    [00:8364] handling command SMPD_ABORT_JOB src=2
    [00:8364] Suspending rank 0, smpd id = 1, ctx_key=0
    [00:8364] posting command SMPD_SUSPEND to left child, src=0, dest=1.
    [00:8364] Suspending rank 1, smpd id = 2, ctx_key=0
    [00:8364] posting command SMPD_SUSPEND to left child, src=0, dest=2.
    [00:8364] Handling cmd=SMPD_SUSPEND result
    [00:8364] cmd=SMPD_SUSPEND result will be handled locally
    [00:8364] Handling cmd=SMPD_SUSPEND result
    [00:8364] cmd=SMPD_SUSPEND result will be handled locally
    [00:8364] posting kill command smpd id=1, ctx_key=0
    [00:8364] posting command SMPD_KILL to left child, src=0, dest=1.
    [00:8364] posting kill command smpd id=2, ctx_key=0
    [00:8364] posting command SMPD_KILL to left child, src=0, dest=2.
    [00:8364] handling command SMPD_EXIT src=1
    [00:8364] Handling cmd=SMPD_KILL result
    [00:8364] cmd=SMPD_KILL result will be handled locally
    [00:8364] handling command SMPD_EXIT src=2
    
    job aborted:
    [ranks] message
    
    [0] terminated
    
    [1] application aborted
    aborting MPI_COMM_WORLD (comm=0x44000000), error 250, comm rank 1
    
    ---- error analysis -----
    
    [1] on 192.168.2.2
    mpipingpong aborted the job. abort code 250
    
    ---- error analysis -----
    [00:8364] last process exited, tearing down the job tree num_exited=2 num_procs=2.
    [00:8364] Handling cmd=SMPD_KILL result
    [00:8364] cmd=SMPD_KILL result will be handled locally
    [00:8364] handling command SMPD_CLOSED src=1
    [00:8364] closed command received from left child.
    [00:8364] smpd manager successfully stopped listening.
    
    C:\>pause
    Press any key to continue . . .

    Costas

    Monday, April 23, 2018 7:46 PM
  • Hello,

    I have also tried to test mpipingpong with smpd instead of MSMPILaunchSvc. I have started smpd.exe listening on port 5100. Here is the error message:

    mpiexec.exe -p 5100 -pwd <pass> -savecreds -hosts 2 DESKTOP-C61LVC8 DESKTOP-MG0GUD5 -env MSMPI_ND_PORT_RANGE 5555,5555 -env MPICH_NETMASK 192.168.2.0/255.255.255.0 -env MSMPI_DISABLE_ND 0 -env MSMPI_DISABLE_SOCK 1 -env MSMPI_CONNECTIVITY_TABLE 1 -affinity -priority 1 mpipingpong -r -op -pc
    
    [ DESKTOP-MG0GUD5#1 ] Fatal Error: MPI Failure
    [ DESKTOP-MG0GUD5#1 ] Error Details:
    [ DESKTOP-MG0GUD5#1 ]   Other MPI error, error stack:
    [ DESKTOP-MG0GUD5#1 ]   MPI_Allgather(sbuf=0x000000000107FDA0, scount=128, MPI_CHAR, rbuf=0x0000000001761B00, rcount=128, MPI_CHAR, MPI_COMM_WORLD) failed
    [ DESKTOP-MG0GUD5#1 ]   unable to connect to 192.168.2.1 192.168.0.15 DESKTOP-C61LVC8  on port 0, the socket interconnect is disabled
    
    
    job aborted:
    [ranks] message
    
    [0] fatal error
    Fatal error in MPI_Allgather: Other MPI error, error stack:
    MPI_Allgather(sbuf=0x000000000067F860, scount=128, MPI_CHAR, rbuf=0x0000000000D51B00, rcount=128, MPI_CHAR, MPI_COMM_WORLD) failed
    [ch3:nd] Could not connect via NetworkDirect to rank 1 with business card (port=0 description="192.168.2.2 192.168.0.24 DESKTOP-MG0GUD5 " shm_host=DESKTOP-MG0GUD5 shm_queue=8116:288 ).
    There is no NetworkDirect information in the business card and the socket interconnect is disabled.
    Check the remote NetworkDirect configuration or clear the MPICH_DISABLE_SOCK environment variable.
    
    [1] application aborted
    aborting MPI_COMM_WORLD (comm=0x44000000), error 250, comm rank 1
    
    ---- error analysis -----
    
    [0] on DESKTOP-C61LVC8
    mpi has detected a fatal error and aborted mpipingpong
    
    ---- error analysis -----

    I have been trying everything I can think of without success.  I would appreciate any feedback.

    Regards,

    Costas

    Thursday, June 7, 2018 8:04 AM