MS-MPI v7 issue with VPN

All replies

• 

  

  0

  Sign in to vote

  When you said it did not work with VPN, did you mean it worked when the VPN was turned off and stopped working when the VPN was turned on? In either case, can you run mpiexec -d 3 -n 2 myApp.exe and paste the output?

  Thanks,

  Anh

  Saturday, January 9, 2016 12:21 AM
• 

  

  0

  Sign in to vote

  Thanks for reply, Anh.

  The case is it worked when the VPN was turned off and stopped working when the VPN was turned on. I can post the log later because I can not use VPN now.

  Jack

  
  

  
  

  • Edited by JACK_CH Saturday, January 9, 2016 12:30 AM typo

  Saturday, January 9, 2016 12:30 AM
• 

  

  0

  Sign in to vote

  The log for the failed case,

  "C:\Program Files\Microsoft MPI\Bin\mpiexec.exe" -n 1 -debug 3 myApp.exe

  [00:9608] host tree:

  [00:9608] host: CARBON, parent: 0, id: 1

  [00:9608] mpiexec started smpd manager listening on port 53610

  [00:9608] create manager process (using mpiexec credentials)

  [00:9608] Command C:\Program Files\Microsoft MPI\Bin\smpd.exe "C:\Program Files\Microsoft MPI\Bin\smpd.exe" -p 8677 -d 11 -mgr 340 "job" -localonly

  [00:9608] smpd reading the port string from the manager

  [-1:9492] Launching smpd manager instance.

  [-1:9492] created set for manager listener, 188

  [-1:9492] smpd manager listening on port 53611

  [-1:9492] manager writing port back to smpd.

  [00:9608] closing the pipe to the manager

  [00:9608] CARBON posting a re-connect to CARBON:53611 in left child context.

  Aborting: mpiexec on CARBON is unable to connect to the smpd manager on localhost:53611 error 5

  [00:9608] ERROR: Failed to connect to SMPD Manager Instance error 5

  [00:9608] smpd manager successfully stopped listening.

  Saturday, January 9, 2016 12:58 AM
• 

  

  0

  Sign in to vote

  It has been a long time since I use AnyConnect and right now I don't have any environment with Anyconnect that I can poke around. Can you check the options/properties of the VPN connection (if you are allowed to configure the change) and see if you have an option that will enable/disable the forwarding of all user traffic over VPN. It might be called split-tunneling in some cases. Meanwhile, I'll continue digging around to see if I can gain access to an environment with AnyConnect running to reproduce the issue

  Thanks

  Anh

  
  

  • Edited by Anh.VoMicrosoft employee Tuesday, January 12, 2016 11:54 PM

  Saturday, January 9, 2016 1:25 AM
• 

  

  0

  Sign in to vote

  Jack,

  Can you also try one more thing

  Start "smpd -d 3" as daemon

  Assuming the name of the computer is CARBON, run mpiexec -host CARBON -n 1 myApp.exe

  When you use localhost, the connection of mpiexec and the instance smpd goes through the loop back adapter. On the other hand, when you specify the name, it goes to the NIC.

  Thanks,

  Anh

  Saturday, January 9, 2016 2:10 AM
• 

  

  0

  Sign in to vote

  Hi Anh, I know the workaround to add a host option and start a smpd daemon, but I do not know how to make the command without a host option, mpiexec -n 1 myApp.exe, work with MS-MPI v7 when VPN is turned on. It used to work with MS-MPI v6.
  

  • Edited by JACK_CH Monday, January 11, 2016 12:04 PM Typo

  Monday, January 11, 2016 12:03 PM
• 

  

  0

  Sign in to vote

  Hi Jack,

  My question wasn't meant to suggest a workaround. Rather, I would like to check whether the issue has to do with loopback or not. We make improvements and changes to MS-MPI between versions and in MS-MPI v7 the protocol for process management changed. Some changes might require some configuration or firewall changes from the users. Your data will help us address the issue.

  If you can start an "smpd -d 3" daemon and provide the output of the command "mpiexec -host CARBON -n 1 myApp.exe" it would help us narrow down the issue.

  Also, can you provide a screenshot of the anyconnect Vpn setting (feel free to edit out any information you do not want to share)?

  Thanks

  Anh

  Monday, January 11, 2016 9:05 PM
• 

  

  0

  Sign in to vote

  The output for command, "mpiexec -host CARBON -n -1 myApp.exe" with running smpd daemon "smpd -d 3",

  ERROR: Failed RpcCliCreateContext error 5
  
  Aborting: mpiexec on CARBON is unable to connect to the smpd service on CARBON
  NN:8677
  Other MPI error, error stack:
  connect failed - Access is denied.  (errno 5)

  
  

  It seems I can not upload images.

  Tuesday, January 12, 2016 8:21 PM
• 

  

  0

  Sign in to vote

  The Preferences page for anyConenct,

  x Start VPN wen AnyConnect is started

  x Minimize AnyConnect on VPN connect

    Allow local(LAN) access then using VPN(if configured)

  x Block connections to untrusted servers

  The Statistics page,

  Connection Information-----------------

  state:

  Tunnel Mode (IPv4): Split Include

  Tunnel Mode (IPv6): Drop All Traffic

  Duration: 00:14:56

  Address Information---------------------

  Client(IPv4): 192.168.3.21

  Client(IPv6): Not Available

  Server: xxx.xxx.208.251

  ... ...

  Anh, if you need more VPN configurations, please let me know.

  Jack

  
  

  
  

  Tuesday, January 12, 2016 9:01 PM
• 

  

  0

  Sign in to vote

  Jack, if you check the box that says "Allow local (LAN) access when using VPN (if configured)", will it work? Note that the VPN configuration might disable it. Let us know

  Tuesday, January 12, 2016 10:45 PM
• 

  

  0

  Sign in to vote

  Hi Anh, after I reconnect my VPN, "Allow local (LAN) access when using VPN (if configured)" is disappeared. It's seems I can not change that option.

  Tuesday, January 12, 2016 11:07 PM
• 

  

  0

  Sign in to vote

  It probably means the VPN configuration is not allowing LAN when using VPN. Let's try another thing.

  If you're in a position to compile and run a simple program can you please run this program and give us the output of the program (while the VPN is connected)

  #include <stdio.h>
  #include <stdlib.h>
  #include <Winsock2.h>

  #pragma comment(lib, "Ws2_32.lib")

  int main()
  {
      char szPath[MAX_COMPUTERNAME_LENGTH + 1] = "";

      WSADATA wsaData;
      WSAStartup(MAKEWORD(2, 2), &wsaData);

      gethostname(szPath, sizeof(szPath));
      printf("gethostname returns %s\n", szPath);

      DWORD size = _countof(szPath);
      GetComputerNameA(szPath, &size);
     
      printf("GetComputerName returns %s\n", szPath);
      WSACleanup();
      return 0;
  }

  Thanks

  Anh

  
  

  • Edited by Anh.VoMicrosoft employee Tuesday, January 12, 2016 11:36 PM

  Tuesday, January 12, 2016 11:32 PM
• 

  

  0

  Sign in to vote

  No mater VPN is connected or not, the output is same,

  gethostname returns CARBON

  GetComputerName returns CARBON
  

  Tuesday, January 12, 2016 11:55 PM
• 

  

  0

  Sign in to vote

  Thanks for running the program and providing us with the output. It ruled out one possible change from v6 to v7. I'll test out a few things and will post a reply soon

  Wednesday, January 13, 2016 12:03 AM
• 

  

  0

  Sign in to vote

  Jack,

  Can you check with your administrator (who configured the VPN that you're connecting to) if they explicitly disable RPC (Remote Procedure Call). The process management protocol for MSMPI v7 relies on RPC and when using the computer's name (vs localhost) the RPC calls will have to go through the VPN. If your VPN configuration blocks all RPC traffic this will cause issue.

  Thanks

  Anh

  Wednesday, January 13, 2016 12:19 AM
• 

  

  0

  Sign in to vote

  Anh,

  Our administrator said that we didn't explicitly block RPC. Do you have any suggestion to confirm it?

  Jack

  Wednesday, January 13, 2016 1:23 AM
• 

  

  0

  Sign in to vote

  Jack,

  I've managed to create an environment where I can reproduce your problem. It's interesting that I'm observing the problem only with the AnyConnect VPN Secure Mobility Client, and not with the older Windows 7 AnyConnect VPN.

  It looks like this is a DNS suffix resolving issue where Windows is trying to resolve your machine name using the remote DNS instead of the local DNS. I'm looking at options on how to change the network configurations to fix this and will post back when I get a good working solution. In the mean time, the following workaround will unblock you

  Edit your hosts file under C:\windows\system32\drivers\etc\hosts (assuming your windows installation is in drive C) and add the following line

  127.0.0.1 CARBON

  Basically this routes the access to your local machine (from within your machine) to the loopback adapter instead of going through the network, which avoids the DNS suffix issue.

  Anh

  
  

  • Edited by Anh.VoMicrosoft employee Wednesday, January 13, 2016 4:46 PM
  • Proposed as answer by Anh.VoMicrosoft employee Tuesday, February 9, 2016 11:31 PM

  Wednesday, January 13, 2016 4:43 PM
• 

  

  0

  Sign in to vote

  Hi Anh,

  Thanks for your work.

  I wonder if the fix will be included in the next release. Do you have a timeframe for the next release? 

  Cheers,

  Jack

  
  

  Friday, January 29, 2016 1:02 AM
• 

  

  0

  Sign in to vote

  Hi Jack,

  The next release will include a fix for this issue. We're planning for a minor release sometime soon. Let me know if you're interested in beta-testing it to let us know if the issue has been resolved for you. We would appreciate it very much. If you would like to do that, please send us an email at askmpi@microsoft.com and include a link to this thread and we'll get back to you through email with instructions on how to obtain a pre-release for testing purposes.

  Thanks!

  Anh

  Tuesday, February 9, 2016 11:31 PM
• 

  

  0

  Sign in to vote

  Edit your hosts file under C:\windows\system32\drivers\etc\hosts (assuming your windows installation is in drive C) and add the following line
  

  127.0.0.1 CARBON

  Basically this routes the access to your local machine (from within your machine) to the loopback adapter instead of going through the network, which avoids the DNS suffix issue.

  Anh

  
  

  Hi Anh,

  I have exact the same problem like Jack. I use MS-MPI v7.0.12437.6 and Cisco AnyConnect Secure Mobility Client Version 3.1.10010. I can activate the local LAN access, but that isn't working at all. Your hint with editing the hosts-file won't work for me.

  If you need a beta tester I will help.Any idea when an update comes?

  Update: I also tried it with Cisco AnyConnect Secure Mobility Client Version 4.2.01035 and it didn't work.

  
  

  • Edited by dommo Thursday, April 7, 2016 4:58 PM

  Thursday, April 7, 2016 10:09 AM
• 

  

  0

  Sign in to vote

  Edit your hosts file under C:\windows\system32\drivers\etc\hosts (assuming your windows installation is in drive C) and add the following line
  

  127.0.0.1 CARBON

  Basically this routes the access to your local machine (from within your machine) to the loopback adapter instead of going through the network, which avoids the DNS suffix issue.

  Anh

  
  

  Hi Anh,

  I have exact the same problem like Jack. I use MS-MPI v7.0.12437.6 and Cisco AnyConnect Secure Mobility Client Version 3.1.10010. I can activate the local LAN access, but that isn't working at all. Your hint with editing the hosts-file won't work for me.

  If you need a beta tester I will help.Any idea when an update comes?

  Update: I also tried it with Cisco AnyConnect Secure Mobility Client Version 4.2.01035 and it didn't work.

  
  

  Hi Dommo,

  The pre-release of v7.1 is now available. If you could give it a try and let us know if the issue with the Cisco Anyconnect VPN is fixed.

  https://www.microsoft.com/en-us/download/details.aspx?id=52042

  Thanks

  
  

  • Edited by Anh.VoMicrosoft employee Wednesday, May 4, 2016 10:06 PM

  Wednesday, May 4, 2016 8:13 PM
• 

  

  0

  Sign in to vote

  Hi Anh,

  Thank you for your reminder for the pre-release.

  I tested it and my issue has been fixed, but my work around is not work any more. I am not sure if it's a bug I should report here.

  The failed case is when I add -host option to the command and vpn is connected,

  mpiexec -host localhost -n 2 myApp.exe

  The error message is,

  ERROR: Failed to connect to SMPD Manager Instance error 1726

  Aborting: mpiexec on CARBON is unable to connect to the smpd manager on localhost:49808 error 1726

  Cheers,

  Jack

  Tuesday, May 10, 2016 12:59 AM
• 

  

  0

  Sign in to vote

  Hi jack,

  Is there still a 127.0.0.1 CARBON host entry in the hosts file?

  What happens if you try to do mpiexec -host CARBON -n 2 myApp.exe ? (i.e., replace localhost with the name of the machine).

  Can you also give me the dump of smpd -d 3 in the failed case and the output of mpiexec -d 3 -host localhost -n 2 myApp.exe

  Thanks

  Tuesday, May 10, 2016 2:15 PM
• 

  

  0

  Sign in to vote

  Is there still a 127.0.0.1 CARBON host entry in the hosts file?
  

  No.

  What happens if you try to do mpiexec -host CARBON -n 2 myApp.exe ? (i.e., replace localhost with the name of the machine).

  Not working. The log is,

  mpiexec -d 3 -host CARBON -n 2 myApp.exe
  [00:6688] host tree:
  [00:6688]  host: CARBON, parent: 0, id: 1
  [00:6688] mpiexec started smpd manager listening on port 52878
  [00:6688] using spn msmpi/PORTMANN to contact server
  [00:6688] Previous attempt failed with error 5, trying to authenticate without Kerberos
  [00:6688] ERROR: Failed RpcCliCreateContext error 5
  
  Aborting: mpiexec on CARBON is unable to connect to the smpd service on CARBON:8677
  Other MPI error, error stack:
  connect failed - Access is denied.  (errno 5)
  [00:6688] smpd manager successfully stopped listening.

  Can you also give me the dump of smpd -d 3 in the failed case and the output of mpiexec -d 3 -host localhost -n 2 myApp.exe

  smpd -d 3
  [-1:3936] Launching SMPD service.
  [-1:3936] smpd listening on port 8677
  [-1:3936] Authentication completed. Successfully obtained Context for Client.
  [-1:3936] version check complete, using PMP version 3.
  [-1:3936] create manager process (using smpd daemon credentials)
  [-1:3936] smpd reading the port string from the manager
  [-1:4164] Launching smpd manager instance.
  [-1:4164] created set for manager listener, 112
  [-1:4164] smpd manager listening on port 52887
  [-1:3936] closing the pipe to the manager
  [-1:4164] Authentication completed. Successfully obtained Context for Client.
  [-1:4164] Authorization completed.
  [-1:4164] version check complete, using PMP version 3.
  [-1:4164] Received session header from parent id=1, parent=0, level=0
  [01:4164] Connecting back to parent using host CARBON and endpoint 52885
  [01:4164] Previous attempt failed with error 5, trying to authenticate without K
  erberos
  [01:4164] Failed to connect back to parent error 5.
  [01:4164] ERROR: Failed to connect back to parent 'ncacn_ip_tcp:CARBON:52885'
  error 5
  [01:4164] smpd manager successfully stopped listening.
  [01:4164] SMPD exiting with error code 4294967293.

  mpiexec -d 3 -host localhost -n 2 myApp.exe
  [00:3532] host tree:
  [00:3532]  host: localhost, parent: 0, id: 1
  [00:3532] mpiexec started smpd manager listening on port 52885
  [00:3532] using spn msmpi/localhost to contact server
  [00:3532] CARBON posting a re-connect to localhost:52887 in left child context
  .
  [00:3532] ERROR: Failed to connect to SMPD Manager Instance error 1726
  
  Aborting: mpiexec on CARBON is unable to connect to the smpd manager on localhost:52887 error 1726
  [00:3532] smpd manager successfully stopped listening.

  
  

  
  
  

  • Edited by JACK_CH Tuesday, May 10, 2016 6:26 PM

  Tuesday, May 10, 2016 5:41 PM
• 

  

  0

  Sign in to vote

  Hi Jack,

  Sorry for the delay in responding, it took a while for me to get access to a reproduce environment. We don't have a native AnyConnect VPN environment here so it has been a bit tricky :-).

  I think I finally figured out the issue. I was able to reproduce exactly the symptom you were seeing. Is there any chance your VPN configuration looks similar to this ?

  

  What happens in this case is that the VPN is configured to drop all IPv6 traffic. In MSMPI v6 and before that we used Winsock and specified that only Ipv4 would be supported and thus things were working fine for VPN environments that only allows ipv4.

  Since MSMPI v7 we moved our process management to RPC, which supports both Ipv6 and Ipv4 (Unfortunately, I don't know of anyway to force RPC to use only ipv4). Basically your hostname carbon is resolved to an Ipv6 address and is blocked by the VPN. In my case a simple test like "ping MYMACHINE" would fail when the vpn is connected. If I try "ping -4 MYMACHINE" it would work. 

  To get it to work I simply disabled ipv6 on the interface (see my screenshot) and things started working

  

  Tuesday, May 24, 2016 5:45 AM