none
unable to change subscribed cores via cmdlet or gui RRS feed

  • Question

  • Server and client are HPC 2016 update 2 with the fixes. 5.2.6291. I am running power shell as administrator. The node is offline. I am a cluster administrator however Set-HPCNode fails.

    PS C:\Windows\system32> Add-PSSnapin Microsoft.HPC
    PS C:\Windows\system32> set-hpcnode -name "mynodename" -subscribedcores "40"
    set-hpcnode : Retry Count of RetryManager is exhausted.
    At line:1 char:1
    + set-hpcnode -name "mynodename" -subscribedcores "40"
    + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
        + CategoryInfo          : NotSpecified: (Microsoft.Compu...tHpcNodeCommand:SetHpcNodeCommand) [Set-HpcNode], Retry
       CountExhaustException
        + FullyQualifiedErrorId : Microsoft.ComputeCluster.CCPPSH.SetHpcNodeCommand

    If I use the cluster administrator GUI, I am unable to change the subscribed cores, the field is just grayed out.


    Sunday, May 19, 2019 8:34 PM

All replies

  • Hi,

    In the cluster administrator GUI, only the subscribed cores filed grayed out or "Edit Properties" button for the node is grayed out?

    And could you run the powershell cmd as below to get more information about the error?

    try {
    set-hpcnode -name "mynodename" -subscribedcores "40" -EA Stop
    }
    catch
    {
        $_.Exception | format-list -Force
    }

    Monday, May 20, 2019 7:51 PM
  • Hello,

    I am able to select edit properties, only the subscribed cores and subscribed sockets is grayed out.

    I attempted to run the command in power shell. Here is the out put.

    Message        : Retry Count of RetryManager is exhausted.
    Data           : {}
    InnerException : System.Net.Http.HttpRequestException: An error occurred while sending the request. --->
                     System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a
                     receive. ---> System.ComponentModel.Win32Exception: The client and server cannot communicate, because
                     they do not possess a common algorithm
                        at System.Net.SSPIWrapper.AcquireCredentialsHandle(SSPIInterface SecModule, String package,
                     CredentialUse intent, SecureCredential scc)
                        at System.Net.Security.SecureChannel.AcquireCredentialsHandle(CredentialUse credUsage,
                     SecureCredential& secureCredential)
                        at System.Net.Security.SecureChannel.AcquireClientCredentials(Byte[]& thumbPrint)
                        at System.Net.Security.SecureChannel.GenerateToken(Byte[] input, Int32 offset, Int32 count,
                     Byte[]& output)
                        at System.Net.Security.SecureChannel.NextMessage(Byte[] incoming, Int32 offset, Int32 count)
                        at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest
                     asyncRequest)
                        at System.Net.Security.SslState.ForceAuthentication(Boolean receiveFirst, Byte[] buffer,
                     AsyncProtocolRequest asyncRequest)
                        at System.Net.Security.SslState.ProcessAuthentication(LazyAsyncResult lazyResult)
                        at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext,
                     ContextCallback callback, Object state, Boolean preserveSyncCtx)
                        at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback
                     callback, Object state, Boolean preserveSyncCtx)
                        at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback
                     callback, Object state)
                        at System.Net.TlsStream.ProcessAuthentication(LazyAsyncResult result)
                        at System.Net.TlsStream.BeginWrite(Byte[] buffer, Int32 offset, Int32 size, AsyncCallback
                     asyncCallback, Object asyncState)
                        at System.Net.ConnectStream.WriteHeaders(Boolean async)
                        --- End of inner exception stack trace ---
                        at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
                        at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
                        --- End of inner exception stack trace ---
                        at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                        at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                        at Microsoft.Hpc.HttpClientExtension.<>c__DisplayClass5_0.<<GetHttpApiCallAsync>b__0>d.MoveNext()
                     --- End of stack trace from previous location where exception was thrown ---
                        at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                        at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                        at Microsoft.Hpc.RetryManager.<InvokeWithRetryAsync>d__33`1.MoveNext()
    TargetSite     : Void MoveNext()
    StackTrace     :    at Microsoft.Hpc.RetryManager.<InvokeWithRetryAsync>d__33`1.MoveNext()
                     --- End of stack trace from previous location where exception was thrown ---
                        at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                        at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                        at Microsoft.Hpc.HttpClientExtension.<GetHttpApiCallAsync>d__5.MoveNext()
                     --- End of stack trace from previous location where exception was thrown ---
                        at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                        at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                        at Microsoft.Hpc.HpcFabricRestContext.<>c__DisplayClass14_1`1.<<GetFirstHttpApiCallResponseAsync>b_
                     _0>d.MoveNext()
                     --- End of stack trace from previous location where exception was thrown ---
                        at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                        at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                        at Microsoft.Hpc.HpcFabricRestContext.<GetFirstHttpApiCallResponseAsync>d__14`1.MoveNext()
                     --- End of stack trace from previous location where exception was thrown ---
                        at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                        at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                        at Microsoft.Hpc.HpcFabricRestContext.<ResolveSingletonServicePrimaryAsync>d__10.MoveNext()
                     --- End of stack trace from previous location where exception was thrown ---
                        at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                        at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                        at Microsoft.Hpc.ServiceResolverExtension.<ResolveSdmNodeAsync>d__18.MoveNext()
                     --- End of stack trace from previous location where exception was thrown ---
                        at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                        at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                        at Microsoft.SystemDefinitionModel.Store.SdmRemoteStore.SdmRemoteStoreConnector.<ConnectAsync>d__2.
                     MoveNext()
                     --- End of stack trace from previous location where exception was thrown ---
                        at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                        at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                        at Microsoft.SystemDefinitionModel.Store.SdmRemoteStore.<ConnectAsync>d__58.MoveNext()
                     --- End of stack trace from previous location where exception was thrown ---
                        at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                        at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                        at Microsoft.SystemDefinitionModel.Store.SdmRemoteStore.CheckConnection()
                        at
                     Microsoft.SystemDefinitionModel.Store.SdmRemoteStore.<>c__DisplayClass45_0.<QueryDocuments>b__0()
                        at Microsoft.SystemDefinitionModel.Store.SdmRetry.<>c__DisplayClass4_0`1.<InvokeWithRetry>b__0()
                        at Microsoft.Hpc.RetryManager.<InvokeWithRetryAsync>d__33`1.MoveNext()
                     --- End of stack trace from previous location where exception was thrown ---
                        at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                        at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                        at Microsoft.SystemDefinitionModel.Store.SdmRetry.InvokeWithRetry[T](Func`1 function,
                     CancellationToken cancellationToken)
                        at Microsoft.SystemDefinitionModel.ModelDocuments.LoadAllDocuments()
                        at Microsoft.SystemDefinitionModel.DefinitionSpaceView.TryResolve(String simpleName)
                        at Microsoft.SystemDefinitionModel.ModelDocuments..ctor(Model model)
                        at Microsoft.SystemDefinitionModel.Model.InitializeModel()
                        at Microsoft.ComputeCluster.Management.ClusterManager.ConnectCore(Model model)
                        at System.Threading.Tasks.Task.Execute()
                     --- End of stack trace from previous location where exception was thrown ---
                        at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                        at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                        at Microsoft.Hpc.RetryManager.<>c__DisplayClass34_0.<<InvokeWithRetryAsync>b__0>d.MoveNext()
                     --- End of stack trace from previous location where exception was thrown ---
                        at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                        at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                        at Microsoft.Hpc.RetryManager.<InvokeWithRetryAsync>d__33`1.MoveNext()
                     --- End of stack trace from previous location where exception was thrown ---
                        at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                        at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                        at Microsoft.Hpc.RetryManager.<InvokeWithRetryAsync>d__34.MoveNext()
                     --- End of stack trace from previous location where exception was thrown ---
                        at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                        at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                        at Microsoft.ComputeCluster.Management.ClusterManager.<ConnectAsync>d__90.MoveNext()
                     --- End of stack trace from previous location where exception was thrown ---
                        at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                        at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                        at Microsoft.ComputeCluster.Management.ClusterManager.<ConnectAsyncWithModel>d__89.MoveNext()
                     --- End of stack trace from previous location where exception was thrown ---
                        at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
                        at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
                        at Microsoft.ComputeCluster.CCPPSH.ConnectionPool.GetClusterManager(String scheduler)
                        at Microsoft.ComputeCluster.CCPPSH.NodeEditCmd.RetrieveNodes()
                        at Microsoft.ComputeCluster.CCPPSH.SetHpcNodeCommand.InternalEndProcessing()
                        at Microsoft.ComputeCluster.CCPPSH.CcpCommand.EndProcessing()
    HelpLink       :
    Source         : HpcCommon
    HResult        : -2146233088

    PS C:\Windows\system32>

    Thank you!

    Monday, May 20, 2019 9:45 PM
  • So, you can connect to the head node via Cluster Manager Console, but failed to run HPC PowerShell Cmdlet in the same client machine, right?
    There can be several reasons for the error "The client and server cannot communicate, because they do not possess a common algorithm". One possible reason is that the TLS version used in the client is different from that on the head node.
    You can use the following PowerShell command line to check the TLS version in the PowerShell console in the client machine.
    [Net.ServicePointManager]::SecurityProtocol
    If it is SSL3,TLS, you can try to modify it as below and retry the HPC powershell command in the same console
    [Net.ServicePointManager]::SecurityProtocol = "Tls, Tls11, Tls12"

    The fields of subscribed cores and sockets in the "Edit Node Properties" dialog are only disabled when the node is not in Offline or UnDeployed State. It doesn't make sense the fields are grayed out when node is in offline state, could you double check the node state?
    Tuesday, May 21, 2019 6:10 PM
  • Thank you for the reply. I ran [Net.ServicePointManager]::SecurityProtocol on both the head node and the client node.

    Both returned the same out put.

    [Net.ServicePointManager]::SecurityProtocol
    Ssl3, Tls

    I ran [Net.ServicePointManager]::SecurityProtocol = "Tls, Tls11, Tls12" then I was able to update the Subscribed cores.

    I am curious if this same issue could be causing other applications to fail to connect. We have a lot of users that use MATLAB to submit jobs. Our head node and compute nodes are all using HPC 2016 update 2 with the latest fix pack. We have found that they can validate through MATLAB using the HPC 2012 client but it fails to connect if HPC 2016 is used. I am wondering if the api calls MATLAB uses to connect to HPC are running in to the same problem.

    Thank you,

    Amber

    Tuesday, May 21, 2019 10:50 PM