HPC dependent tasks only on success

23 августа 2011 г. 5:43

0

Is there any way to have dependant tasks that are only executed on the success of the task they depend on?

It seems like a basic requirement that if i want a task to only execute if a prior task succeeds but the documentation says:

Typically, you use dependencies to break a large task into smaller subtasks. For example, if task A needs to finish before task B, which needs to finish before task C, then task B depends on A and task C depends on B. Task B will not run until A finishes, fails, or is canceled.
- Ответить
- Цитировать

Все ответы

10 сентября 2011 г. 20:08

Hi,

Unfortunatelly there is no built-in way of doing this, however it can be accomplished with a simple workaround. You can wrap execution of the dependant task to a Powershell script, which will connect back to the scheduler to check the status of executed dependencies and it will either fail current task or run provided commandline.

Here's an example:

#
# Single parameter to provide a commandline
# to be executed if all dependencies are
# finished succesfully
#
param([string] $Commandline)

#
# Load HPC cmdlets
#
Add-PSSnapin Microsoft.HPC

#
# Get currently executed task, CCP_JOBID and CCP_TASKID
# environment variables are provided by HPCNodeManager
#
$currentTask = Get-HPCTask -JobId $env:CCP_JOBID -TaskId $env:CCP_TASKID

#
# Extract array of dependency names from 
# comma delimited string property
$depNames = $currentTask.Depend.Split(',');

#
# Get collection of all tasks in a current job filtered to only contain:
# 1) Tasks with names from the dependencies list
# 2) Tasks in states other than 'Finished'
#
$depUnsuccesful = Get-HPCTask -JobId $env:CCP_JOBID `
     | Where { $depNames -contains $_.Name -and $_.State -ne "Finished" }

#
# Check if returned collection is empty,
# if not fail current task, otherwise
# execute provided commandline
#
if($depUnsuccesful -ne $null)
{
    "Detected unsuccesful task dependencies:"
    $depUnsuccesful
    ""
    "Aborting current task execution."
    exit -1
}
else
{
    Invoke-Expression $Commandline
}

There are some limitations of such approach, like:
- task dependencies have to be set with use of specific task names provided via API, CLI or Powershell interfaces, so this will not work with task groups created via GUI,
- Powershell execution policy has to be set appropriately run this script from a remote locations.

I tested above script by saving it to c:\tests\Run-HPCDependantTask.ps1 file on all my computenodes. Then I created job with the following script:

$job = new-hpcjob

add-hpctask -jobid $job.Id -Commandline "hostname" -Name "A"
add-hpctask -jobid $job.Id -Commandline "hostname" -Name "B"
add-hpctask -jobid $job.Id -Commandline "invalid_command" -Name "C"
add-hpctask -jobid $job.Id -Commandline "invalid_command" -Name "D"
add-hpctask -jobid $job.Id -Commandline "powershell c:\tests\Run-HPCDependantTask.ps1 -Commandline 'ping -n 2 localhost'" -Name "test1" -Depend "A,B,C,D"
add-hpctask -jobid $job.Id -Commandline "powershell c:\tests\Run-HPCDependantTask.ps1 -Commandline 'ping -n 2 localhost'" -Name "test2" -Depend "A,B"
add-hpctask -jobid $job.Id -Commandline "powershell c:\tests\Run-HPCDependantTask.ps1 -Commandline 'ping -n 2 localhost'" -Name "test3" -Depend "A"
add-hpctask -jobid $job.Id -Commandline "powershell c:\tests\Run-HPCDependantTask.ps1 -Commandline 'ping -n 2 localhost'" -Name "test4" -Depend "C"

Submit-HpcJob $job.Id

Tasks A and B will succeed and tasks C and D will fail. Tasks test1 and test4 will not execute their 'Commandline', because of dependency failures. Instead they will list failed dependencies and fail immediatelly.

Here's the example output of test1 task:

Detected unsuccesful task dependencies:

Id       Name       State           CommandLine               ErrorMessage     
--       ----       -----           -----------               ------------     
3        C          Failed          invalid_command           Task failed du...
4        D          Failed          invalid_command           Task failed du...

Aborting current task execution.

Please let me know if you find provided information useful or if you have any questions.

Regards,
Łukasz