locked
How to cancel jobs that keep being submitted by BrokerClient.SendRequest()? RRS feed

  • Question

  • Hello,

    By using a PowerShell script invoked from a C# client application, I can cancel HPC jobs running on all the compute nodes with the Get-HpcJob and Stop-HPCJob cmdlets run on the head node through Invoke-Command.

    The problem is that this will only take care of jobs running on the compute nodes at a certain point in time, not those that keep being submitted by BrokerClient.SendRequest() from the C# client.

    Is there a way to cancel HPC jobs at the Microsoft.HPC.Scheduler.Session level where they come from?

    Thank you.

    Marc





    • Edited by MarcSim Monday, August 8, 2016 4:34 PM
    Monday, August 8, 2016 1:43 AM

Answers

  • Hi Marc,

    Once you have canceled the SOA job, all the compute node resource would be released for other jobs. The broker worker on the broker node, which is responsible for receiving the requests from the client and dispatching them to the service hosts on the compute nodes for processing,  would detect the service job has been canceled and close itself automatically. Once the broker worker is closed, the client won't be able to send requests to the broker. So there is no worry that any compute resource would be wasted after the session job is canceled. To be more safe, you may explicitly call SessionBase.Close to close the session when canceling the service job.

    Regards,

    Yutong Sun

    • Marked as answer by MarcSim Tuesday, August 9, 2016 9:17 PM
    Tuesday, August 9, 2016 8:18 AM

All replies

  • Hi,

      I might not be quite understand your scenario. For Get-HPCJob and Stop-HPCJob cmdlets, they are treating SOA/Session Job the same as other jobs (Batch and MPI job). Thus you can stop those sessions jobs without any problems. Just that the BrokerClient need to handle the situation when their job is cancelled.


    Qiufang Shi

    Tuesday, August 9, 2016 12:50 AM
  • Hello Qiufang Shi,

    Please let me try to clarify my scenario.

    My users asked to be able to cancel long simulations from my client C# application. This is because they might have realized that they have uploaded the wrong input file so they want to prevent hours or days of useless calculations to be performed by the Azure cluster.

    I implemented such a capability via a Cancel button in the C# application interface, which invokes a PowerShell script. Basically, this script makes use of Get-HPCJob (to retrieve the job IDs) and Stop-HPCJob and this works fine ... by connecting to the head node via RDP, I can see in real time that the jobs are canceled quickly. So this part is fine.

    The problem is that after the Cancel button has been clicked, other jobs can be submitted to the Azure cluster by the asynchronous module in the C# application responsible for sending requests. My understanding is that those jobs will not be canceled, they will consume resources and cost money. This is what I want to avoid. I guess I would have to intervene on the .NET Session object to "stop" it, but I do not know how to do that.

    Thanks again.

    Marc

    • Marked as answer by MarcSim Tuesday, August 9, 2016 9:16 PM
    • Unmarked as answer by MarcSim Tuesday, August 9, 2016 9:16 PM
    Tuesday, August 9, 2016 1:19 AM
  • Hi Marc,

    Once you have canceled the SOA job, all the compute node resource would be released for other jobs. The broker worker on the broker node, which is responsible for receiving the requests from the client and dispatching them to the service hosts on the compute nodes for processing,  would detect the service job has been canceled and close itself automatically. Once the broker worker is closed, the client won't be able to send requests to the broker. So there is no worry that any compute resource would be wasted after the session job is canceled. To be more safe, you may explicitly call SessionBase.Close to close the session when canceling the service job.

    Regards,

    Yutong Sun

    • Marked as answer by MarcSim Tuesday, August 9, 2016 9:17 PM
    Tuesday, August 9, 2016 8:18 AM
  • Hello Yutong Sun,

    Thank you very much for your detailed answer, which fully addresses my concern.

    Regards,

    Marc

    Tuesday, August 9, 2016 9:18 PM