none
connect failed - Access is denied. (errno 5)

    Question

  • Hello!

    I’m trying to run Microsoft’s CNTK across multiple Windows computers using MPIEXEC as described in the link.

    I’m able to run mpiexec on my computer, e.g.:

    mpiexec -hosts 1 MY_COMPUTER 4  python ae_cntk.py

    But it doesn’t work if I wish to connect to other computer:

    mpiexec -hosts 2 MY_COMPUTER 4 OTHER_COMPUTER 4 python ae_cntk.py

    smpd –d is running on both MY_COMPUTER and OTHER_COMPUTER.

    Error:

    Aborting: smpd on MY_COMPUTER is unable to connect to the smpd service on OTHER_COMPUTER:8677

    Other MPI error, error stack:

    connect failed - Access is denied.  (errno 5)

    What is wrong with MY_COMPUTER?

    Best Regards,

    Arijit

    PS:

    Mpiexec between OTHER_COMPUTER and ANOTHER_COMPUTER works! So, mpiexec works in general, but I do not understand why it doesn’t work on MY_COMPUTER.

    Wednesday, September 13, 2017 2:51 PM

All replies

  • Hi Arijit,

    Are you using the same username/password on MY_COMPUTER and OTHER_COMPUTER? Also, which version of MS-MPI are you using (you can get the version by running mpiexec -help and look at the first few lines of the output)

    Anh

    Wednesday, September 13, 2017 10:02 PM
  • Hi Anh:

    Both MY_COMPUTER and OTHER_COMPUTER are Windows 10. I'm not using the same username/password. 

    I'm using MS-MPI version

    Version 7.0.12437.6 in both the computers. 

    Best Regards,

    Arijit


    Thursday, September 14, 2017 7:53 AM
  • Hi Arijit,

    You will need to use the same account with same user name and password for both machines and start smpd -d as well as mpiexec using that account for MPI to work across those two machines. In addition, I would recommend using the latest version of MS-MPI, which is v8.1

    Anh

    Thursday, September 14, 2017 2:45 PM
  • Hi Anh:

    Thanks.

    Is there no other way?

    I'll give it a try if it works with same username and password. 

    Also, I cannot use v8.1, because Microsoft CNTK recommends using version 7 https://docs.microsoft.com/en-us/cognitive-toolkit/Multiple-GPUs-and-machines

    Best regards,

    Arijit

    Friday, September 15, 2017 10:34 AM
  • Hi Ariji,

    Unfortunately, there is no workaround for the requirement of having same user (in a domain network) or same username/password.

    You should not have any issue using v8.1 with CNTK as MS-MPI is backward compatible

    Monday, September 18, 2017 12:19 AM
  • Hi Anh:

    I updated to MS-MPI version 8.1.12438.1084.

    (As I wrote in my first e-mail: mpiexec between OTHER_COMPUTER and ANOTHER_COMPUTER works à updated MS-MPI works with CNTK!)

    But like before, still there is problem between MY_COMPUTER and OTHER_COMPUTER. Both computers now use same username and password.

    If I try to run mpiexec from MY_COMPUTER, I see this error:

    ERROR: Failed RpcCliCreateContext error 5

    Aborting: mpiexec on MY_COMPUTER is unable to connect to the smpd service on OTHER_COMPUTER:8677

    Other MPI error, error stack:

    connect failed - Access is denied.  (errno 5)

    If I try to run mpiexec from OTHER_COMPUTER, I see this error:

    Aborting: smpd on OTHER_COMPUTER is unable to connect to the smpd manager on MY_COMPUTER:63595 error 1726

    On OTHER_COMPUTER smpd, I see

    [01:28060] ERROR: Failed to connect to SMPD Manager Instance error 1726

    [01:28060] ERROR: smpd running on OTHER_COMPUTER is unable to connect to smpd service on MY_COMPUTER:8677

    On MY_COMPUTER smpd, I see

    [02:7500] ERROR: Failed to connect back to parent 'ncacn_ip_tcp: OTHER_COMPUTER:6073' error 5

    8 hours 10 minutes ago