locked
HPC Pack 2016 will not connect to HPC 2012 Head Node RRS feed

  • Question

  • Hello,

    I have a Server 2016 PC with HPC Pack 2016 5.2.6277.0 installed.  When I open the HPC Cluster Manager and try to connect to a Server 2012 Head Node with version HPC Pack 2012 4.5.5079.0 installed, I get the following error message:

    ***************************************************

    The connection to the management service failed. detail error: Microsoft.Hpc.RetryCountExhaustException: Retry Count of RetryManager is exhausted. ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a send. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
       at System.Net.Sockets.Socket.BeginReceive(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, AsyncCallback callback, Object state)
       at System.Net.Sockets.NetworkStream.BeginRead(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback callback, Object state)
       --- End of inner exception stack trace ---
       at System.Net.TlsStream.EndWrite(IAsyncResult asyncResult)
       at System.Net.ConnectStream.WriteHeadersCallback(IAsyncResult ar)
       --- End of inner exception stack trace ---
       at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
       at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
       --- End of inner exception stack trace ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.HttpClientExtension.<>c__DisplayClass5_0.<<GetHttpApiCallAsync>b__0>d.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.RetryManager.<InvokeWithRetryAsync>d__33`1.MoveNext()
       --- End of inner exception stack trace ---
       at Microsoft.Hpc.RetryManager.<InvokeWithRetryAsync>d__33`1.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.HttpClientExtension.<GetHttpApiCallAsync>d__5.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.HpcFabricRestContext.<>c__DisplayClass14_1`1.<<GetFirstHttpApiCallResponseAsync>b__0>d.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.HpcFabricRestContext.<GetFirstHttpApiCallResponseAsync>d__14`1.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.HpcFabricRestContext.<ResolveSingletonServicePrimaryAsync>d__10.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.ServiceResolverExtension.<ResolveSdmNodeAsync>d__18.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.SystemDefinitionModel.Store.SdmRemoteStore.SdmRemoteStoreConnector.<ConnectAsync>d__2.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.SystemDefinitionModel.Store.SdmRemoteStore.<ConnectAsync>d__58.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.SystemDefinitionModel.Store.SdmRemoteStore.CheckConnection()
       at Microsoft.SystemDefinitionModel.Store.SdmRemoteStore.<>c__DisplayClass45_0.<QueryDocuments>b__0()
       at Microsoft.SystemDefinitionModel.Store.SdmRetry.<>c__DisplayClass4_0`1.<InvokeWithRetry>b__0()
       at Microsoft.Hpc.RetryManager.<InvokeWithRetryAsync>d__33`1.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.SystemDefinitionModel.Store.SdmRetry.InvokeWithRetry[T](Func`1 function, CancellationToken cancellationToken)
       at Microsoft.SystemDefinitionModel.ModelDocuments.LoadAllDocuments()
       at Microsoft.SystemDefinitionModel.DefinitionSpaceView.TryResolve(String simpleName)
       at Microsoft.SystemDefinitionModel.ModelDocuments..ctor(Model model)
       at Microsoft.SystemDefinitionModel.Model.InitializeModel()
       at Microsoft.ComputeCluster.Management.ClusterManager.ConnectCore(Model model)
       at System.Threading.Tasks.Task.Execute()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.RetryManager.<>c__DisplayClass34_0.<<InvokeWithRetryAsync>b__0>d.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.RetryManager.<InvokeWithRetryAsync>d__33`1.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.RetryManager.<InvokeWithRetryAsync>d__34.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.ComputeCluster.Management.ClusterManager.<ConnectAsync>d__90.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.ComputeCluster.Management.ClusterManager.<ConnectAsyncWithModel>d__89.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.ComputeCluster.Management.ClusterManager.<ConnectAsync>d__87.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.ComputeCluster.Admin.ConnectionManager.<ConnectManagementService>b__10_0(Task`1 t)

    **************************************************************************

    According to the below link, from HPC Pack 2016 I should be able to connect to a 2012 Head Node and submit jobs.

    https://docs.microsoft.com/en-us/powershell/high-performance-computing/what-s-new-in-hpc-pack-2016-update-1?view=hpc16-ps

    Thanks,

    Tiffany

    Thursday, December 13, 2018 3:49 PM

All replies

  • Hi Tiffany,

      You shall use HPC Job Manager instead of HPC Cluster Manager. The backward compatibility is target for job.


    Qiufang Shi

    Friday, December 14, 2018 3:25 AM
  • Thank you, that worked.

    Friday, December 14, 2018 2:38 PM
  • I'm still having issues when it comes to connecting to the scheduler.  We have a C# app that uses the IScheduler.Connect() method and we want to be able to run it from a HPC Server 2016 connecting to a HPC Server 2012 R2 installation but its not working.

    Tuesday, January 8, 2019 3:09 PM
  • What error did you get when connecting to a HPC Server 2012 R2?

    If HPCJobManager can connect to HPC Server 2012 R2, your app shall be able to as well.


    Qiufang Shi

    Wednesday, January 9, 2019 6:44 AM
  • The error message itself states that : one or more errors has occurred.

    The  error Source is : mscorlib.

    The error stacktrace is:

       at Microsoft.Hpc.Scheduler.Store.StoreServer.<InternalConnectAsync>d__38.MoveNext()

    --- End of stack trace from previous location where exception was thrown ---

       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

       at Microsoft.Hpc.Scheduler.Store.StoreServer.<ConnectAsync>d__34.MoveNext()

    --- End of stack trace from previous location where exception was thrown ---

       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

       at Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.<InitializeAsync>d__41.MoveNext()

    --- End of stack trace from previous location where exception was thrown ---

       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

       at Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.<RemoteConnectAsync>d__4.MoveNext()

    --- End of stack trace from previous location where exception was thrown ---

       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

       at Microsoft.Hpc.Scheduler.Store.SchedulerStore.<ConnectAsync>d__1.MoveNext()

    --- End of stack trace from previous location where exception was thrown ---

       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()

       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

       at Microsoft.Hpc.Scheduler.Scheduler.<ConnectAsync>d__108.MoveNext()

    Monday, January 14, 2019 2:37 PM