none
HPC Pack 2016 VM Azure Cluster. Scheduler.Connect Error when called via Web API hosted on IIS RRS feed

  • Question

  • I set up a Azure HPC 2016 VM Cluster.   I have C# web api running on the IIS of the Headnode acting as a client to submit the jobs to HPC Cluster.  IIS application pool (where the web api was hosted) running on custom domain account.  When calling the Scheduler.Connect the below error is being thrown

    ExecptionMessage - Inner Exception: One or more errors occurred. Inner Exception: The operation was canceled by the user
    ExceptionStackFrame - Inner StackTrace:    at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
       at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationToken cancellationToken)
       at Microsoft.Hpc.Scheduler.Scheduler.Connect(String cluster, ConnectMethod method)
    Inner StackTrace:    at Microsoft.Hpc.Credentials.PromptForCredentials(String target, String& username, SecureString& password, Boolean& fSave, Boolean fConsole, IntPtr hwndParent, CREDUI_FLAGS flags, String message)
       at Microsoft.Hpc.Scheduler.Store.StoreServer.<InternalConnectOverHttpAsync>d__36.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.Scheduler.Store.StoreServer.<ConnectAsync>d__34.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.<InitializeAsync>d__41.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.<RemoteConnectAsync>d__4.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.Scheduler.Store.SchedulerStore.<ConnectAsync>d__1.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.Scheduler.Scheduler.<ConnectAsync>d__108.MoveNext()

    I am trying to connect to the HPC scheduler to submit the JOBS under the custom domain account that was configured on the IIS. I have added this custom domain user account to HPC Job Operators group but it is still throwing the error. 

    It appears IIS request to call the Scheduler.Connect under the domain account was being blocked for some reason which I have no clue. Any suggestions or ideas to resolve this issue? I have been trying to resolve this issue for last couple of days.  The piece of code is working fine on OnPremise HPC cluster. 

                    IScheduler scheduler = new Scheduler();
                    utl.Logger.LogMessage("Before connecting", Logging.Category.Information);
                    scheduler.Connect("https://mycloud.centralus.cloudapp.azure.com", ConnectMethod.WCF);
                    ISchedulerJob job = scheduler.CreateJob();

     Thanks in advance

    Ramesh

    Saturday, May 11, 2019 10:42 AM

All replies

  • Hi Ramesh,

    You shall run the following command as the domain account which you use to run IIS application pool.

    hpcred setcreds

    After that, the domain account can submit job without input credential.

    More details about this command please see https://docs.microsoft.com/en-us/powershell/high-performance-computing/hpccred-setcreds?view=hpc16-ps

    Wednesday, May 15, 2019 6:28 AM
  • Thank you Sunbin for your reply. 

    In fact I have tried this earlier but it didn't worked either. 

    At the HPC Power Shell, below is the command I did against the domain user.

    hpccred setcreds /scheduler:https://****.******.cloudapp.azure.com /user:<domain>\hpcadmin /password:********

    Then I executed the below command to verify whether creds were cached, but 

    hpccred getcreds /scheduler:https://****.******.cloudapp.azure.com 

    At the HPC Powershell command promot,  the message is " no softcard credentials saved for the user on the server".

    Regardless, I have tried to launch my ASP.NET application to verify whether I am able to connect to the Scheduler , but I did encountered the same error as above.

    I have configured the IIS Default App Pool for the same specified domain user to submit the jobs to the cluster when the web page is requested. 

    Is there anything else I am missing.  Highly appreciate if you have any other suggestions to get past this issue. 

    Many Thanks

    Ramesh

    Thursday, May 16, 2019 5:28 AM
  • Where did you run this command? 

    Since your Web application (which acts as the HPC client) is hosted on the IIS of the head node, you shall run the command on the head node.

    And you don't need to specify the '/scheduler' option.

    And why you use FQDN of the head node when connecting itself? you can just use hostname instead.

    Saturday, May 18, 2019 5:52 PM