locked
HPC Session Service is stopped sometimes when Windows Server 2016 is rebooted RRS feed

  • Question

  • Hello,

    We are using Microsoft HPC Pack 2016 Update 3 on Windows Server 2016. We noticed that "HPC Session Service" is sometimes stopped when the server is rebooted.

    The workaround we use is to start it from the Services app, after which, the port 9094 is open and is listening for incoming requests.

    We noticed that this service is stopped only sometimes and not on all server reboots. All these reboots were usual and nothing special was installed or no updates were run during reboot.

    Below is a Trace fragement when the service was stopped after server reboot.
    """"
    08/19/2020 10:16:03.147 i HpcSession.exe 1216 1204 [GetCertificateValidationCallback] Bypass certificate CN validation.  
    08/19/2020 10:16:03.147 i HpcSession.exe 1216 1204 [GetCertificateValidationCallback] Bypass certificate CN validation.  
    08/19/2020 10:16:04.601 v HpcSoa 1216 4188 Before connect service  
    08/19/2020 10:16:09.620 i HpcSession.exe 1216 4188 [WcfProxy] Begin connect to endpointStr net.tcp://<servername>:5802/SchedulerStoreServiceInternal, dnsIdentityName HPC Pack 2016 Communication  
    08/19/2020 10:16:11.595 i HpcSession.exe 1216 4188 [WcfProxy] End to create internal wcf proxy  
    08/19/2020 10:16:16.540 w HpcSession.exe 1216 4188 [WcfProxy] Channel to net.tcp://<servername>:5802/SchedulerStoreServiceInternal faulted.  
    08/19/2020 10:16:17.255 e HpcSoa 1216 4188 [CommonSchedulerHelper]. GetScheduler: System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://<servername>:5802/SchedulerStoreServiceInternal. The connection attempt lasted for a time span of 00:00:02.1135733. TCP error code 10061: No connection could be made because the target machine actively refused it [::1]:5802.  ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it [::1]:5802..   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)..   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)..   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)..   --- End of inner exception stack trace ---....Server stack trace: ..   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)..   at System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout)..   at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)..   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)..   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)..   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)..   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)..   at System.ServiceModel.Channels.ServiceChannel.CallOpenOnce.System.ServiceModel.Channels.ServiceChannel.ICallOnce.Call(ServiceChannel channel, TimeSpan timeout)..   at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)..   at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)..   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)..   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)..   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)....Exception rethrown at [0]: ..   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)..   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)..   at Microsoft.Hpc.Scheduler.Store.ISchedulerStoreInternal.Register(String clientSource, String userName, ConnectionRole role, Version clientVersion, ConnectionToken& token, UserPrivilege& privilege, Version& serverVersion, Dictionary`2& serverProps)..   at Microsoft.Hpc.Scheduler.Store.StoreServer.RegisterWithServer()..   at Microsoft.Hpc.Scheduler.Store.StoreServer.RegisterEvent(String schedulerNode)..   at Microsoft.Hpc.Scheduler.Store.StoreServer.<ConnectWcfAsync>d__39.MoveNext()..--- End of stack trace from previous location where exception was thrown ---..   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()..   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)..   at Microsoft.Hpc.Scheduler.Store.StoreServer.<InternalConnectAsync>d__38.MoveNext()..--- End of stack trace from previous location where exception was thrown ---..   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()..   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)..   at Microsoft.Hpc.Scheduler.Store.StoreServer.<ConnectAsync>d__34.MoveNext()..--- End of stack trace from previous location where exception was thrown ---..   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()..   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)..   at Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.<InitializeAsync>d__41.MoveNext()..--- End of stack trace from previous location where exception was thrown ---..   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()..   a  
    08/19/2020 10:16:19.325 v HpcSoa 1216 4196 Before connect service  
    08/19/2020 10:16:19.325 i HpcSession.exe 1216 4196 [WcfProxy] Begin connect to endpointStr net.tcp://<servername>:5802/SchedulerStoreServiceInternal, dnsIdentityName HPC Pack 2016 Communication  
    08/19/2020 10:16:19.325 i HpcSession.exe 1216 4196 [WcfProxy] End to create internal wcf proxy  
    08/19/2020 10:16:21.773 w HpcSession.exe 1216 4196 [WcfProxy] Channel to net.tcp://<servername>:5802/SchedulerStoreServiceInternal faulted.  
    08/19/2020 10:16:21.790 e HpcSoa 1216 4196 [CommonSchedulerHelper]. GetScheduler: System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://<servername>:5802/SchedulerStoreServiceInternal. The connection attempt lasted for a time span of 00:00:02.4184476. TCP error code 10061: No connection could be made because the target machine actively refused it [::1]:5802.  ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it [::1]:5802..   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)..   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)..   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)..   --- End of inner exception stack trace ---....Server stack trace: ..   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)..   at System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout)..   at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)..   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)..   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)..   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)..   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)..   at System.ServiceModel.Channels.ServiceChannel.CallOpenOnce.System.ServiceModel.Channels.ServiceChannel.ICallOnce.Call(ServiceChannel channel, TimeSpan timeout)..   at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)..   at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)..   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)..   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)..   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)....Exception rethrown at [0]: ..   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)..   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)..   at Microsoft.Hpc.Scheduler.Store.ISchedulerStoreInternal.Register(String clientSource, String userName, ConnectionRole role, Version clientVersion, ConnectionToken& token, UserPrivilege& privilege, Version& serverVersion, Dictionary`2& serverProps)..   at Microsoft.Hpc.Scheduler.Store.StoreServer.RegisterWithServer()..   at Microsoft.Hpc.Scheduler.Store.StoreServer.RegisterEvent(String schedulerNode)..   at Microsoft.Hpc.Scheduler.Store.StoreServer.<ConnectWcfAsync>d__39.MoveNext()..--- End of stack trace from previous location where exception was thrown ---..   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()..   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)..   at Microsoft.Hpc.Scheduler.Store.StoreServer.<InternalConnectAsync>d__38.MoveNext()..--- End of stack trace from previous location where exception was thrown ---..   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()..   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)..   at Microsoft.Hpc.Scheduler.Store.StoreServer.<ConnectAsync>d__34.MoveNext()..--- End of stack trace from previous location where exception was thrown ---..   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()..   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)..   at Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.<InitializeAsync>d__41.MoveNext()..--- End of stack trace from previous location where exception was thrown ---..   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()..   a  
    08/19/2020 10:16:25.867 v HpcSoa 1216 4196 Before connect service  
    08/19/2020 10:16:25.879 i HpcSession.exe 1216 4196 [WcfProxy] Begin connect to endpointStr net.tcp://<servername>:5802/SchedulerStoreServiceInternal, dnsIdentityName HPC Pack 2016 Communication  
    08/19/2020 10:16:25.879 i HpcSession.exe 1216 4196 [WcfProxy] End to create internal wcf proxy  
    08/19/2020 10:16:28.092 w HpcSession.exe 1216 4196 [WcfProxy] Channel to net.tcp://<servername>:5802/SchedulerStoreServiceInternal faulted.  
    08/19/2020 10:16:28.092 e HpcSoa 1216 4196 [CommonSchedulerHelper]. GetScheduler: System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://<servername>:5802/SchedulerStoreServiceInternal. The connection attempt lasted for a time span of 00:00:02.2135825. TCP error code 10061: No connection could be made because the target machine actively refused it [::1]:5802.  ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it [::1]:5802..   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)..   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)..   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)..   --- End of inner exception stack trace ---....Server stack trace: ..   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)..   at System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout)..   at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)..   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)..   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)..   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)..   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)..   at System.ServiceModel.Channels.ServiceChannel.CallOpenOnce.System.ServiceModel.Channels.ServiceChannel.ICallOnce.Call(ServiceChannel channel, TimeSpan timeout)..   at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)..   at System.ServiceModel.Channels.ServiceChannel.EnsureOpened(TimeSpan timeout)..   at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)..   at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)..   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)....Exception rethrown at [0]: ..   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)..   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)..   at Microsoft.Hpc.Scheduler.Store.ISchedulerStoreInternal.Register(String clientSource, String userName, ConnectionRole role, Version clientVersion, ConnectionToken& token, UserPrivilege& privilege, Version& serverVersion, Dictionary`2& serverProps)..   at Microsoft.Hpc.Scheduler.Store.StoreServer.RegisterWithServer()..   at Microsoft.Hpc.Scheduler.Store.StoreServer.RegisterEvent(String schedulerNode)..   at Microsoft.Hpc.Scheduler.Store.StoreServer.<ConnectWcfAsync>d__39.MoveNext()..--- End of stack trace from previous location where exception was thrown ---..   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()..   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)..   at Microsoft.Hpc.Scheduler.Store.StoreServer.<InternalConnectAsync>d__38.MoveNext()..--- End of stack trace from previous location where exception was thrown ---..   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()..   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)..   at Microsoft.Hpc.Scheduler.Store.StoreServer.<ConnectAsync>d__34.MoveNext()..--- End of stack trace from previous location where exception was thrown ---..   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()..   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)..   at Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.<InitializeAsync>d__41.MoveNext()..--- End of stack trace from previous location where exception was thrown ---..   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()..   a  
    08/19/2020 10:16:36.165 v HpcSoa 1216 4196 Before connect service  
    08/19/2020 10:16:36.165 i HpcSession.exe 1216 4196 [WcfProxy] Begin connect to endpointStr net.tcp://<servername>:5802/SchedulerStoreServiceInternal, dnsIdentityName HPC Pack 2016 Communication  
    08/19/2020 10:16:36.165 i HpcSession.exe 1216 4196 [WcfProxy] End to create internal wcf proxy  
    08/19/2020 10:16:38.292 w HpcSession.exe 1216 4196 [WcfProxy] Channel to net.tcp://<servername>:5802/SchedulerStoreServiceInternal faulted.  
    08/19/2020 10:16:38.292 e HpcSoa 1216 4196 [CommonSchedulerHelper]. GetScheduler: System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://<servername>:5802/SchedulerStoreServiceInternal. The connection attempt lasted for a time span of 00:00:02.1173140. TCP error code 10061: No connection could be made because the target machine actively refused it [::1]:5802.  ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it [::1]:5802..   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)..   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)..   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)..   --- End of inner exception stack trace ---....Server stack trace: ..   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)..   at System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout)..   at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)..   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)..   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)..   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)..   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)..   at System.ServiceModel.Channels.ServiceChannel.CallOpenOnce.System.ServiceModel.Channels.ServiceChannel.ICallOnce.Call(ServiceChannel channel, TimeSpan timeout)..   at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)..   at System.ServiceModel.Channels.ServiceChannel.Ensur֏.θ  TimeSpan timeout)..   at System. ’w",    €󎐁     s.Se                l(String                       J›ϸ  erationR               ˆ’w",           Object[] outs, TimeSpan time        at ¨       erviceMo        nels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)..   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)....Exception rethrown at [0]: ..   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)..   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)..   at Microsoft.Hpc.Scheduler.Store.ISchedulerStoreInternal.Register(String clientSource, String userName, ConnectionRole role, Version clientVersion, ConnectionToken& token, UserPrivilege& privilege, Version& serverVersion, Dictionary`2& serverProps)..   at Microsoft.Hpc.Scheduler.Store.StoreServer.RegisterWithServer()..   at Microsoft.Hpc.Scheduler.Store.StoreServer.RegisterEvent(String schedulerNode)..   at Microsoft.Hpc.Scheduler.Store.StoreServer.<ConnectWcfAsync>d__39.MoveNext()..--- End of stack trace from previous location where exception was thrown ---..   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()..   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)..   at Microsoft.Hpc.Scheduler.Store.StoreServer.<InternalConnectAsync>d__38.MoveNext()..--- End of stack trace from previous location where exception was thrown ---..   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()..   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)..   at Microsoft.Hpc.Scheduler.Store.StoreServer.<ConnectAsync>d__34.MoveNext()..--- End of stack trace from previous location where exception was thrown ---..   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()..   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)..   at Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.<InitializeAsync>d__41.MoveNext()..--- End of stack trace from previous location where exception was thrown ---..   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()..   a  
    08/19/2020 10:16:54.373 v HpcSoa 1216 5888 Before connect service  
    08/19/2020 10:16:54.373 i HpcSession.exe 1216 5888 [WcfProxy] Begin connect to endpointStr net.tcp://<servername>:5802/SchedulerStoreServiceInternal, dnsIdentityName HPC Pack 2016 Communication  
    08/19/2020 10:16:54.373 i HpcSession.exe 1216 5888 [WcfProxy] End to create internal wcf proxy  
    08/19/2020 10:16:56.726 w HpcSession.exe 1216 5888 [WcfProxy] Channel to net.tcp://<servername>:5802/SchedulerStoreServiceInternal faulted.  
    08/19/2020 10:16:56.726 e HpcSoa 1216 5888 [CommonSchedulerHelper]. GetScheduler: System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://<servername>:5802/SchedulerStoreServiceInternal. The connection attempt lasted for a time span of 00:00:02.3532395. TCP error code 10061: No connection could be made because the target machine actively refused it [::1]:5802.  ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it [::1]:5802..   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)..   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)..   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)..   --- End of inner exception stack trace ---....Server stack trace: ..   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)..   at System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout)..   at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)..   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)..   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)..   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)..   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)..   at System.ServiceModel.Channels.ServiceChannel.CallOpenOnce.System.ServiceModel.Channels.ServiceChannel.ICallOnce.Call(ServiceChannel channel, TimeSpan timeout)..   at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)..   at System.ServiceModel.Channels.ServiceChannel.Ensur֏.θ  TimeSpan timeout)..   at System. ’w",    €󎐁     s.Se                l(String                       J›ϸ  erationR               ˆ’w",           Object[] outs, TimeSpan time        at ¨       erviceMo        nels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)..   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)....Exception rethrown at [0]: ..   at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)..   at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)..   at Microsoft.Hpc.Scheduler.Store.ISchedulerStoreInternal.Register(String clientSource, String userName, ConnectionRole role, Version clientVersion, ConnectionToken& token, UserPrivilege& privilege, Version& serverVersion, Dictionary`2& serverProps)..   at Microsoft.Hpc.Scheduler.Store.StoreServer.RegisterWithServer()..   at Microsoft.Hpc.Scheduler.Store.StoreServer.RegisterEvent(String schedulerNode)..   at Microsoft.Hpc.Scheduler.Store.StoreServer.<ConnectWcfAsync>d__39.MoveNext()..--- End of stack trace from previous location where exception was thrown ---..   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()..   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)..   at Microsoft.Hpc.Scheduler.Store.StoreServer.<InternalConnectAsync>d__38.MoveNext()..--- End of stack trace from previous location where exception was thrown ---..   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()..   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)..   at Microsoft.Hpc.Scheduler.Store.StoreServer.<ConnectAsync>d__34.MoveNext()..--- End of stack trace from previous location where exception was thrown ---..   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()..   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)..   at Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.<InitializeAsync>d__41.MoveNext()..--- End of stack trace from previous location where exception was thrown ---..   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()..   a  
    08/19/2020 10:17:26.782 e HpcSoa 1216 4188 Failed to open the service host - System.ServiceModel.EndpointNotFoundException: Could not connect to net.tcp://<servername>:5802/SchedulerStoreServiceInternal. The connection attempt lasted for a time span of 00:00:02.3532395. TCP error code 10061: No connection could be made because the target machine actively refused it [::1]:5802.  ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it [::1]:5802..   at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)..   at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)..   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)..   --- End of inner exception stack trace ---....Server stack trace: ..   at System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout)..   at System.ServiceModel.Channels.BufferedConnectionInitiator.Connect(Uri uri, TimeSpan timeout)..   at System.ServiceModel.Channels.ConnectionPoolHelper.EstablishConnection(TimeSpan timeout)..   at System.ServiceModel.Channels.ClientFramingDuplexSessionChannel.OnOpen(TimeSpan timeout)..   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)..   at System.ServiceModel.Channels.ServiceChannel.OnOpen(TimeSpan timeout)..   at System.ServiceModel.Channels.CommunicationObject.Open(TimeSpan timeout)..   at System.ServiceModel.Channels.ServiceChannel.CallOpenOnce.System.ServiceModel.Channels.ServiceChannel.ICallOnce.Call(ServiceChannel channel, TimeSpan timeout)..   at System.ServiceModel.Channels.ServiceChannel.CallOnceManager.CallOnce(TimeSpan timeout, CallOnceManager cascade)..   at System.ServiceModel.Channels.ServiceChannel.EnsureOpen֏.θ  pan timeout)..   at System.Servi ’w",    €󎐁     vice                ing acti                       J›ϸ  onRuntim               ˆ’w",           t[] outs, TimeSpan timeout).       yste¨       eModel.C        ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)..   at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)....Exception rethrown at [0]: ..   at Microsoft.Hpc.Scheduler.Session.Internal.Common.CommonSchedulerHelper.<GetScheduler>d__0.MoveNext()..--- End of stack trace from previous location where exception was thrown ---..   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()..   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)..   at Microsoft.Hpc.Scheduler.Session.Internal.SessionLauncher.BrokerNodesManager..ctor()..   at Microsoft.Hpc.Scheduler.Session.LauncherHostService.LauncherHostService.<OpenService>d__10.MoveNext()  
    08/19/2020 10:17:26.813 v HpcSoa 1216 4188 Session launcher endpoint closed  
    08/19/2020 10:17:26.813 v HpcSoa 1216 4188 Scheduler delegation service closed  
    08/19/2020 10:17:26.836 i HpcTrace 1216 2656 Current Application Domain ProcessExit event invoked  
    08/19/2020 10:17:26.836 i HpcTrace 1216 2656 Cosmos Logger is being closed  
    08/19/2020 10:22:11.777 i HpcSession.exe 388 8344 [GetCertificateValidationCallback] Bypass certificate CN validation.  
    08/19/2020 10:22:11.777 i HpcSession.exe 388 8344 [GetCertificateValidationCallback] Bypass certificate CN validation.  
    08/19/2020 10:22:11.861 v HpcSoa 388 516 Before connect service  
    08/19/2020 10:22:12.010 i HpcSession.exe 388 516 [WcfProxy] Begin connect to endpointStr net.tcp://<servername>:5802/SchedulerStoreServiceInternal, dnsIdentityName HPC Pack 2016 Communication  
    08/19/2020 10:22:12.054 i HpcSession.exe 388 516 [WcfProxy] End to create internal wcf proxy  
    """"
    Is it a known issue that was fixed in a later version?

    If not, how can this issue be resolved?

    Thanks,

    SriMP

    Wednesday, September 2, 2020 1:22 PM

Answers

  • Hi,

    As a workaround, you can set Startup type of HpcSession service from Automatic to Automatic (Delayed Start)

    • Marked as answer by SriMP Monday, September 7, 2020 9:58 AM
    Friday, September 4, 2020 10:26 AM

All replies

  • Hi,

    As a workaround, you can set Startup type of HpcSession service from Automatic to Automatic (Delayed Start)

    • Marked as answer by SriMP Monday, September 7, 2020 9:58 AM
    Friday, September 4, 2020 10:26 AM
  • Thank you for your answer.  This workaround worked for us.

    Monday, September 7, 2020 9:58 AM