none
MS-MPI v7 issue with VPN RRS feed

  • Question

  • Hi,

    I just upgraded MS-MPI from v6 to v7 for my program, and found v7 is not working with VPN(I used anyConnect) when running it on localhost, but v6 worked well.

    The command,

    mpiexec -n 2 myApp.exe

    OS: windows 7

    I found a workaround to run it in cmd window by adding an option '-host localhost' and opening another cmd window to run 'smpd -d', but no way to make my GUI program work as the way it did with MS-MPI v6.

    Is this a bug just introduced in MS-MPI v7?

    Thanks,

    Jack


    Friday, January 8, 2016 11:25 PM

All replies

  • When you said it did not work with VPN, did you mean it worked when the VPN was turned off and stopped working when the VPN was turned on? In either case, can you run mpiexec -d 3 -n 2 myApp.exe and paste the output?

    Thanks,

    Anh

    Saturday, January 9, 2016 12:21 AM
  • Thanks for reply, Anh.

    The case is it worked when the VPN was turned off and stopped working when the VPN was turned on. I can post the log later because I can not use VPN now.

    Jack



    • Edited by JACK_CH Saturday, January 9, 2016 12:30 AM typo
    Saturday, January 9, 2016 12:30 AM
  • The log for the failed case,

    "C:\Program Files\Microsoft MPI\Bin\mpiexec.exe" -n 1 -debug 3 myApp.exe

    [00:9608] host tree:

    [00:9608] host: CARBON, parent: 0, id: 1

    [00:9608] mpiexec started smpd manager listening on port 53610

    [00:9608] create manager process (using mpiexec credentials)

    [00:9608] Command C:\Program Files\Microsoft MPI\Bin\smpd.exe "C:\Program Files\Microsoft MPI\Bin\smpd.exe" -p 8677 -d 11 -mgr 340 "job" -localonly

    [00:9608] smpd reading the port string from the manager

    [-1:9492] Launching smpd manager instance.

    [-1:9492] created set for manager listener, 188

    [-1:9492] smpd manager listening on port 53611

    [-1:9492] manager writing port back to smpd.

    [00:9608] closing the pipe to the manager

    [00:9608] CARBON posting a re-connect to CARBON:53611 in left child context.

    Aborting: mpiexec on CARBON is unable to connect to the smpd manager on localhost:53611 error 5

    [00:9608] ERROR: Failed to connect to SMPD Manager Instance error 5

    [00:9608] smpd manager successfully stopped listening.

    Saturday, January 9, 2016 12:58 AM
  • It has been a long time since I use AnyConnect and right now I don't have any environment with Anyconnect that I can poke around. Can you check the options/properties of the VPN connection (if you are allowed to configure the change) and see if you have an option that will enable/disable the forwarding of all user traffic over VPN. It might be called split-tunneling in some cases. Meanwhile, I'll continue digging around to see if I can gain access to an environment with AnyConnect running to reproduce the issue

    Thanks

    Anh


    Saturday, January 9, 2016 1:25 AM
  • Jack,

    Can you also try one more thing

    Start "smpd -d 3" as daemon

    Assuming the name of the computer is CARBON, run mpiexec -host CARBON -n 1 myApp.exe

    When you use localhost, the connection of mpiexec and the instance smpd goes through the loop back adapter. On the other hand, when you specify the name, it goes to the NIC.

    Thanks,

    Anh

    Saturday, January 9, 2016 2:10 AM
  • Hi Anh, I know the workaround to add a host option and start a smpd daemon, but I do not know how to make the command without a host option, mpiexec -n 1 myApp.exe, work with MS-MPI v7 when VPN is turned on. It used to work with MS-MPI v6.
    • Edited by JACK_CH Monday, January 11, 2016 12:04 PM Typo
    Monday, January 11, 2016 12:03 PM
  • Hi Jack,

    My question wasn't meant to suggest a workaround. Rather, I would like to check whether the issue has to do with loopback or not. We make improvements and changes to MS-MPI between versions and in MS-MPI v7 the protocol for process management changed. Some changes might require some configuration or firewall changes from the users. Your data will help us address the issue.

    If you can start an "smpd -d 3" daemon and provide the output of the command "mpiexec -host CARBON -n 1 myApp.exe" it would help us narrow down the issue.

    Also, can you provide a screenshot of the anyconnect Vpn setting (feel free to edit out any information you do not want to share)?

    Thanks

    Anh

    Monday, January 11, 2016 9:05 PM
  • The output for command, "mpiexec -host CARBON -n -1 myApp.exe" with running smpd daemon "smpd -d 3",

    ERROR: Failed RpcCliCreateContext error 5

    Aborting: mpiexec on CARBON is unable to connect to the smpd service on CARBON
    NN:8677
    Other MPI error, error stack:
    connect failed - Access is denied.  (errno 5)


    It seems I can not upload images.

    Tuesday, January 12, 2016 8:21 PM
  • The Preferences page for anyConenct,

    x Start VPN wen AnyConnect is started

    x Minimize AnyConnect on VPN connect

      Allow local(LAN) access then using VPN(if configured)

    x Block connections to untrusted servers

    The Statistics page,

    Connection Information-----------------

    state:

    Tunnel Mode (IPv4): Split Include

    Tunnel Mode (IPv6): Drop All Traffic

    Duration: 00:14:56

    Address Information---------------------

    Client(IPv4): 192.168.3.21

    Client(IPv6): Not Available

    Server: xxx.xxx.208.251

    ... ...

    Anh, if you need more VPN configurations, please let me know.

    Jack



    Tuesday, January 12, 2016 9:01 PM
  • Jack, if you check the box that says "Allow local (LAN) access when using VPN (if configured)", will it work? Note that the VPN configuration might disable it. Let us know
    Tuesday, January 12, 2016 10:45 PM
  • Hi Anh, after I reconnect my VPN, "Allow local (LAN) access when using VPN (if configured)" is disappeared. It's seems I can not change that option.
    Tuesday, January 12, 2016 11:07 PM
  • It probably means the VPN configuration is not allowing LAN when using VPN. Let's try another thing.

    If you're in a position to compile and run a simple program can you please run this program and give us the output of the program (while the VPN is connected)

    #include <stdio.h>
    #include <stdlib.h>
    #include <Winsock2.h>

    #pragma comment(lib, "Ws2_32.lib")

    int main()
    {
        char szPath[MAX_COMPUTERNAME_LENGTH + 1] = "";

        WSADATA wsaData;
        WSAStartup(MAKEWORD(2, 2), &wsaData);

        gethostname(szPath, sizeof(szPath));
        printf("gethostname returns %s\n", szPath);

        DWORD size = _countof(szPath);
        GetComputerNameA(szPath, &size);
       
        printf("GetComputerName returns %s\n", szPath);
        WSACleanup();
        return 0;
    }

    Thanks

    Anh


    Tuesday, January 12, 2016 11:32 PM
  • No mater VPN is connected or not, the output is same,

    gethostname returns CARBON

    GetComputerName returns CARBON

    Tuesday, January 12, 2016 11:55 PM
  • Thanks for running the program and providing us with the output. It ruled out one possible change from v6 to v7. I'll test out a few things and will post a reply soon
    Wednesday, January 13, 2016 12:03 AM
  • Jack,

    Can you check with your administrator (who configured the VPN that you're connecting to) if they explicitly disable RPC (Remote Procedure Call). The process management protocol for MSMPI v7 relies on RPC and when using the computer's name (vs localhost) the RPC calls will have to go through the VPN. If your VPN configuration blocks all RPC traffic this will cause issue.

    Thanks

    Anh

    Wednesday, January 13, 2016 12:19 AM
  • Anh,

    Our administrator said that we didn't explicitly block RPC. Do you have any suggestion to confirm it?

    Jack

    Wednesday, January 13, 2016 1:23 AM
  • Jack,

    I've managed to create an environment where I can reproduce your problem. It's interesting that I'm observing the problem only with the AnyConnect VPN Secure Mobility Client, and not with the older Windows 7 AnyConnect VPN.

    It looks like this is a DNS suffix resolving issue where Windows is trying to resolve your machine name using the remote DNS instead of the local DNS. I'm looking at options on how to change the network configurations to fix this and will post back when I get a good working solution. In the mean time, the following workaround will unblock you

    Edit your hosts file under C:\windows\system32\drivers\etc\hosts (assuming your windows installation is in drive C) and add the following line

    127.0.0.1 CARBON

    Basically this routes the access to your local machine (from within your machine) to the loopback adapter instead of going through the network, which avoids the DNS suffix issue.

    Anh


    Wednesday, January 13, 2016 4:43 PM
  • Hi Anh,

    Thanks for your work.

    I wonder if the fix will be included in the next release. Do you have a timeframe for the next release? 

    Cheers,

    Jack


    Friday, January 29, 2016 1:02 AM
  • Hi Jack,

    The next release will include a fix for this issue. We're planning for a minor release sometime soon. Let me know if you're interested in beta-testing it to let us know if the issue has been resolved for you. We would appreciate it very much. If you would like to do that, please send us an email at askmpi@microsoft.com and include a link to this thread and we'll get back to you through email with instructions on how to obtain a pre-release for testing purposes.

    Thanks!

    Anh

    Tuesday, February 9, 2016 11:31 PM
  • Edit your hosts file under C:\windows\system32\drivers\etc\hosts (assuming your windows installation is in drive C) and add the following line

    127.0.0.1 CARBON

    Basically this routes the access to your local machine (from within your machine) to the loopback adapter instead of going through the network, which avoids the DNS suffix issue.

    Anh


    Hi Anh,

    I have exact the same problem like Jack. I use MS-MPI v7.0.12437.6 and Cisco AnyConnect Secure Mobility Client Version 3.1.10010. I can activate the local LAN access, but that isn't working at all. Your hint with editing the hosts-file won't work for me.

    If you need a beta tester I will help.Any idea when an update comes?

    Update: I also tried it with Cisco AnyConnect Secure Mobility Client Version 4.2.01035 and it didn't work.


    • Edited by dommo Thursday, April 7, 2016 4:58 PM
    Thursday, April 7, 2016 10:09 AM
  • Edit your hosts file under C:\windows\system32\drivers\etc\hosts (assuming your windows installation is in drive C) and add the following line

    127.0.0.1 CARBON

    Basically this routes the access to your local machine (from within your machine) to the loopback adapter instead of going through the network, which avoids the DNS suffix issue.

    Anh


    Hi Anh,

    I have exact the same problem like Jack. I use MS-MPI v7.0.12437.6 and Cisco AnyConnect Secure Mobility Client Version 3.1.10010. I can activate the local LAN access, but that isn't working at all. Your hint with editing the hosts-file won't work for me.

    If you need a beta tester I will help.Any idea when an update comes?

    Update: I also tried it with Cisco AnyConnect Secure Mobility Client Version 4.2.01035 and it didn't work.


    Hi Dommo,

    The pre-release of v7.1 is now available. If you could give it a try and let us know if the issue with the Cisco Anyconnect VPN is fixed.

    https://www.microsoft.com/en-us/download/details.aspx?id=52042

    Thanks


    Wednesday, May 4, 2016 8:13 PM
  • Hi Anh,

    Thank you for your reminder for the pre-release.

    I tested it and my issue has been fixed, but my work around is not work any more. I am not sure if it's a bug I should report here.

    The failed case is when I add -host option to the command and vpn is connected,

    mpiexec -host localhost -n 2 myApp.exe

    The error message is,

    ERROR: Failed to connect to SMPD Manager Instance error 1726

    Aborting: mpiexec on CARBON is unable to connect to the smpd manager on localhost:49808 error 1726

    Cheers,

    Jack

    Tuesday, May 10, 2016 12:59 AM
  • Hi jack,

    Is there still a 127.0.0.1 CARBON host entry in the hosts file?

    What happens if you try to do mpiexec -host CARBON -n 2 myApp.exe ? (i.e., replace localhost with the name of the machine).

    Can you also give me the dump of smpd -d 3 in the failed case and the output of mpiexec -d 3 -host localhost -n 2 myApp.exe

    Thanks

    Tuesday, May 10, 2016 2:15 PM
  • Is there still a 127.0.0.1 CARBON host entry in the hosts file?

    No.

    What happens if you try to do mpiexec -host CARBON -n 2 myApp.exe ? (i.e., replace localhost with the name of the machine).

    Not working. The log is,

    mpiexec -d 3 -host CARBON -n 2 myApp.exe
    [00:6688] host tree:
    [00:6688]  host: CARBON, parent: 0, id: 1
    [00:6688] mpiexec started smpd manager listening on port 52878
    [00:6688] using spn msmpi/PORTMANN to contact server
    [00:6688] Previous attempt failed with error 5, trying to authenticate without Kerberos
    [00:6688] ERROR: Failed RpcCliCreateContext error 5

    Aborting: mpiexec on CARBON is unable to connect to the smpd service on CARBON:8677
    Other MPI error, error stack:
    connect failed - Access is denied.  (errno 5)
    [00:6688] smpd manager successfully stopped listening.

    Can you also give me the dump of smpd -d 3 in the failed case and the output of mpiexec -d 3 -host localhost -n 2 myApp.exe

    smpd -d 3
    [-1:3936] Launching SMPD service.
    [-1:3936] smpd listening on port 8677
    [-1:3936] Authentication completed. Successfully obtained Context for Client.
    [-1:3936] version check complete, using PMP version 3.
    [-1:3936] create manager process (using smpd daemon credentials)
    [-1:3936] smpd reading the port string from the manager
    [-1:4164] Launching smpd manager instance.
    [-1:4164] created set for manager listener, 112
    [-1:4164] smpd manager listening on port 52887
    [-1:3936] closing the pipe to the manager
    [-1:4164] Authentication completed. Successfully obtained Context for Client.
    [-1:4164] Authorization completed.
    [-1:4164] version check complete, using PMP version 3.
    [-1:4164] Received session header from parent id=1, parent=0, level=0
    [01:4164] Connecting back to parent using host CARBON and endpoint 52885
    [01:4164] Previous attempt failed with error 5, trying to authenticate without K
    erberos
    [01:4164] Failed to connect back to parent error 5.
    [01:4164] ERROR: Failed to connect back to parent 'ncacn_ip_tcp:CARBON:52885'
    error 5
    [01:4164] smpd manager successfully stopped listening.
    [01:4164] SMPD exiting with error code 4294967293.

    mpiexec -d 3 -host localhost -n 2 myApp.exe
    [00:3532] host tree:
    [00:3532]  host: localhost, parent: 0, id: 1
    [00:3532] mpiexec started smpd manager listening on port 52885
    [00:3532] using spn msmpi/localhost to contact server
    [00:3532] CARBON posting a re-connect to localhost:52887 in left child context
    .
    [00:3532] ERROR: Failed to connect to SMPD Manager Instance error 1726

    Aborting: mpiexec on CARBON is unable to connect to the smpd manager on localhost:52887 error 1726
    [00:3532] smpd manager successfully stopped listening.




    • Edited by JACK_CH Tuesday, May 10, 2016 6:26 PM
    Tuesday, May 10, 2016 5:41 PM
  • Hi Jack,

    Sorry for the delay in responding, it took a while for me to get access to a reproduce environment. We don't have a native AnyConnect VPN environment here so it has been a bit tricky :-).

    I think I finally figured out the issue. I was able to reproduce exactly the symptom you were seeing. Is there any chance your VPN configuration looks similar to this ?

    

    What happens in this case is that the VPN is configured to drop all IPv6 traffic. In MSMPI v6 and before that we used Winsock and specified that only Ipv4 would be supported and thus things were working fine for VPN environments that only allows ipv4.

    Since MSMPI v7 we moved our process management to RPC, which supports both Ipv6 and Ipv4 (Unfortunately, I don't know of anyway to force RPC to use only ipv4). Basically your hostname carbon is resolved to an Ipv6 address and is blocked by the VPN. In my case a simple test like "ping MYMACHINE" would fail when the vpn is connected. If I try "ping -4 MYMACHINE" it would work. 

    To get it to work I simply disabled ipv6 on the interface (see my screenshot) and things started working

    Tuesday, May 24, 2016 5:45 AM