locked
dependencies using task names vs task id:job id unique combination? RRS feed

  • Question

  • I've noticed some strange behavior when I am running jobs with FailDependentTasks set to true.

    The first thing that makes me a little nervous is that setting task dependencies is by name instead by task id or task id:job id pair. Are task dependencies always restricted to task names within a single job?

    It seems like a no brain-er but today I launched several of the same job simultaneously. I started to see tasks that failed and they say that they fail because their parent tasks failed. But when I look over the job no parent tasks have failed. However there are task with the same exact name in other currently running jobs that may have failed.

    Right now I am running the jobs with FailDependentTasks set to false. After I will run with FailDependentTasks set to true. Then a third job with FailDependentTasks set to true but I will prefix the job name into the task name so all the simultaneously running tasks are forced to a unique name.

    So the question for now is....behind the scenes... Are task dependency look-ups always restricted to task names within a single job?

    Wednesday, July 20, 2016 9:24 PM

All replies

  • 1. For your first question, task name is used to make it easy for users to create dependencies between tasks. Within the scheduler system, we use task group to describe the task dependencies. If you export the job to xml file, you can see the task group information. Thus:

    • One advantage to use task name instead of task id is to easy the scenario that your task depends on thousands of tasks while you don't need to specify the task id;
    • Help the scenario that a task depend on a group of tasks that may be added later time (We support adding task into running job): "task name" is static
    • And yes, task dependencies restricted to the job scope
    • And of course, You can create your task dependencies with task groups instead. Task groups are in tree structure and a task can only be assigned with one task group

    2. For you second problem, you need check the task group information. If you create your task dependencies with task name, these tasks will be put in one task group, thus the failure reason is correct.

    3. Didn't quite get your third question. And your last question, the answer is yes, the task dependencies are restricted to a single job, and being translated to task groups during job/task validation. And the task group information may change if you add new tasks to the job (Even the job is running).


    Qiufang Shi

    Thursday, July 21, 2016 6:25 AM
  • OK. I think I have three new questions based on testing last night.

    ---

    In the dependency map, I notice the first group id and its first child have no refrence to tasks later in the xml file.

    I assume the first parent groupid 752402 is the job itself? And the first child groupid of 752402 is maybe the node prep task? But what is confusing is that the nodeprep task is defined later in the xml and has a group id of 0.

    What are these 2 groupid's?

    ---

    Setting the FailDependantTasks to false seems to work.

    When I set FailDependantTasks to true I have strange task failures where it says that a parent task has failed....when all task above that task have completed...and when that task has no dependancies set with -depends.

    I am wondering if this is a bug. Is FailDependantTasks is new to the latest version of HPC?

    ---

    The -depends flag in Add-HPCTask seems to be a little brittle. It would be nice when debugging things to be able to have Add-HPCTask return a task object and then be able to create an array of those task objects to use for the -depends flag.

    This would fix two of the immediate gotchas we ran into. Tasks with the same name (or first same 80 characters of the name) used in dependency string obviously don't work right. Tasks with commas in the name would be handled correctly because of the string array seems to be being flatted to a comma separated string behind the scenes .... bad bad bad! :)

    Any way we could have the length of the task names be customizabile to more than 80 characters?

    Any way we could had depends take and array of HPCTask object pointers so the duplicate names and the commas in the name restriction is removed?


    robert jackson

    Friday, July 22, 2016 3:19 PM
  • 1. nodePre/nodeRelease task is different type of tasks. When to run this task is not controlled by task group, but controlled by: run the node pre task when the node is allocated to the job and run the node release task when the node resource is released from the job

    The first group id is the root group, when there is no task dependency created in the job, all tasks will belong to that root group. And when task dependency is created, all task will be moved out this group. So the normal task group graph will looks like this:

    Level 0: ROOT

    Level 1, Task Group Under ROOT: TaskGroup for "TasksWithoutSubTaskGroup+TasksWithoutTaskName", TaskGroup with task name A with SubGroups, ...

    And every Task Group in Level will have Level 2 Task Groups with similar sub groups as level 1. And when a new task is being created, the task will being put in the right task group.

    2. Could you provide me a concrete sample for the issue? We can have a quick check. And this is introduced in V4 (HPC Pack 2012). And we will also have a double check on this.

    3. For the current design, Add-HPCTask will return an HPCJob object so that the operation can be piped like: New-HPCJob|Add-HPCTask|Add-HPCTask|...|Submit-HPCJob

    Looks like it makes you hard to create task dependencies with task names only. We may think of expose the task group object in the task creation which might be able to easy your scenario. Things like below:

    - $job = New-HPCJob

    - $Root = $job.RootGroup

    - $TaskGroup1 = New-HPCTaskGroup -Parent $Root

    - ...

    - New-HPCTask -TaskGroup $TaskGroup?

    And you shall manage the whole dependency correctly.

    4. 80 Char in the task names is a DB schema limitation now. In your case, what makes the name exceeds 80 chars? And what are the safe limit for your task name?

    5. We need evaluate your proposal on taking an array of HPCTask objects.


    Qiufang Shi

    Monday, July 25, 2016 2:02 AM