Determine if a task is being cancelled or preempted RRS feed

  • Question

  • When a task is cancelled the executable receives Control->Break, and has a graceperiod to tidy up before the process exits.

    When a task is preempted the same happens.

    is there anyway for the executable on the computenode to know if the Control->Break is for a cancellation vs pre-emption?

    • Edited by TimJRoberts1 Wednesday, November 29, 2017 4:19 PM
    Wednesday, November 29, 2017 4:13 PM

All replies

  • I think the question you are asking is: how the task process to differentiate the "Ctl+Break" event from a user cancellation or immediate preemption?

    I think from the process point of view, there is no difference and immediate preemption is to cancel a running task. There is only difference at scheduler side, one will requeue the task, one will not.

    You may tell what you what to accomplish and we can check whether there is other way to do it.

    Qiufang Shi

    Friday, December 1, 2017 8:03 AM
  • My command line tasks are pushing their output to a datastore directly from the computenode.

    Consumers interested in the results can get the output directly from the datastore without registering for HPC Scheduler events. 

    If a task is cancelled the task will push any partial results to the data store. The consumer can then read the output from the cancelled task.

    But if a task is pre-empted the same thing happens.

    I have intentionally avoided making the data store aware of HPC.

    I have also intentionally avoided making the job submitter (to HPC scheduler) aware of the datastore.

    All works correctly except for this pre-emption scenario

    Any suggestions? (other than making datastore aware of HPC, or job submitter aware of datastore)?

    Tuesday, December 12, 2017 12:09 PM
  • First, in the upcoming release of HPC Pack 2016 Update 1, there will be a PeekOutput API/Command against a running task, this might be help for your scenario (It will work for linux, windows, azure nodes). And give us the feedbacks if it won't work appropriately for your scenario.

    So what you want is: if the task is cancelled by user, partial results shall go to your datastore, if it is preempted, just clear the partial output from the datastore?

    I believe, what you need to do is:

    1. always put the partial results in the datastore

    2. when task being cancelled (Either user cancel or preempted), append a output indicating the task is being cancelled or preempted.

    3. And actually, your application can get this information from the compute node by running "task view <%CCP_JOBID%>.<%CCP_TASKINSTANCE%?> /detailed" you will see "RequestCancel" value, if it is cancelled by user, it will be marked as "CancelByUser", based on this value, you can decide whether to append more information to the datastore.

    Qiufang Shi

    Wednesday, December 13, 2017 3:16 AM
  • Thanks for your answer. Yes it seems I cannot avoid adding a dependency on HPCScheduler to either the consumer or the datastore.

    When will HPC Pack 2016 Update 1 be available?

    Thursday, December 14, 2017 4:10 PM
  • Currently it is under publishing of the document, but you shall get the bits now 

    Qiufang Shi

    Friday, December 15, 2017 1:26 AM