none
Submit tasks to be executed in a specific order RRS feed

  • Question

  • Hello experts,

    I am new in HPC and in parallel programming and this is the chalenge i am facing.

    I have a sequential code in VB6 (i am old fashioned guy) which includes varius tasks. We would like to see how we can take advantage of HPC and reduce the processing time. At the present time we have a small HPC cluster with one Head node and one Compute node. Each one has 4 cores. Head node is also used as compute node.
    My thought is this: I can split the initial code and build 14 different exe (end to end scenario each one). In order to have the same results with my sequential version these exes should run in this way (c stand for core, t for task):
    c1   c2   c3   c4   c5   c6
    t1   t2
    t3         t4    t5   t6
    t7         t8    t9   t10 t11
    t12       t13
    tj14

    By describing the above: t1 and t2 can start run at the same time. When t1 in finished, t3, t4, t5 and t6 start to run at the same time. When the above are finished, t7, t8, t9, t10 and t11 start to run at the same time. When the above are finished, t12 and t13 start to run at the same time. When t12, t13 AND t2 are finished then t14 run. (c1 c2 etc are indicative. each task can be assigned to the first free core).

    Is this possible to be done through HPC or my whole conceptual approach is in wrong way? I know that a lot of other ways can be more efficient by applying parallel processing and of course translation in other language but at the present time i would prefer something "easy". Another way would be parametric sweep task but let forget it at the moment and focus on this.

    Thank you in advance

    Cheers

    Chris
     
    Edit: I changed the "job" with "task" because i think is the most appropriate keyword for what exactly i want to do after i study some new documents. At the moment i study task dependencies help provided here http://technet.microsoft.com/en-us/library/cc972816%28WS.10%29.aspx If i create 5 Groups of tasks each one including the tasks of each row (see table above), do you think that this will meet my requirements?

    Sunday, December 20, 2009 4:04 PM

Answers

  • Hi Chris,

    Sorry for the confusion over the job add /depend command syntax. You can specify tasknames using a comma-delimited list. The following is an example:

    job add 123 /depend:task1name,task2name notepad.exe

    BTW The reason that specifying /depend:"task1name" "task2name" does not generate an error is that job.exe interprets everything at the end (e.g. "task2name") as being part of the command to be executed.

    Regards,

    Patrick
    Thursday, February 4, 2010 6:35 PM

All replies

  • Hi Chris,

    Yes, it sounds like task dependencies will do exactly what you want with all tasks in the one job. One caveat, though, is that the scheduler does not guarantee that tasks will run at the same time. But using task dependencies will guarantee that a task will not start until all the tasks on which it depends have completed (finished, failed, or canceled).

    Regards,

    Patrick
    Tuesday, December 22, 2009 7:39 PM
  • Hi Patrick and thank you for your answer. Based on your answer 2 more questions came to my mind.

    1) "...the scheduler does not guarantee that tasks will run at the same time." This is based on other submitted jobs? If my job is the only one in the scheduler will the tasks of the same group, start together?

    2)"...have completed (finished, failed, or canceled)". I assume i can control that. i.e. the task of the next group will start only of the tasks of the previous group finished correctly. Is that true?

    Monday, December 28, 2009 9:22 AM
  • Hi Chris,

    [1] In general, yes. But the time at which a task starts is affected by many things, including available resources (cores, sockets, nodes), and network latency. If your job is the only one in the scheduler, and your job has enough resources for all tasks in the same group to start at the same time, then they will likely all start very close to one another in time, but the scheduler does not guarantee startup simultaneity. If a set of tasks absolutely requires that they must all be running simultaneously (e.g. because they must communicate with each other), then the tasks will need to implement this startup-synchronization themselves.

    [2] There is no way to control whether a group of tasks only starts running when tasks in another group are, say, Finished instead of when they are Failed or Canceled. i.e. The dependency is only sensitive to whether all the tasks in a group are completed (Finished, Failed, or Canceled). However, each Job does have a [FailOnTaskFailure] property that, when set, causes the job to fail immediately if any task in the job fails. This would effectively result in no groups of tasks ever running if the groups on which they depend have any Failed tasks.

    Regards,

    Patrick
    Monday, December 28, 2009 9:06 PM
  • Hello again.

    Finally i had my first tries but i met the following problem. My task dependencies are not linear. Therefore i had to create and submit the job through command prompt. Some of my tasks depent on more than one task thus when i was creating each task the "/depend" paramater was like this: /depend:"task1name" "task2name" without any error during job add.
    However when i import it in job manager some errors occurred and when i export the job, the xml file was corrupted. More specifically the lines of the 2 tasks that depend on more that 1 task was a mess.
    So my questions are:
    Is it possible to define a task that depend on more that one task?
    What is the syntax of /depend:<task_name> parameter in job add for more than 1 task?
    What is the syntax of these task in the xml file? I have tried all the below without any success:
    DependsOn="task1" DependsOn="task2" DependsOn="task3" DependsOn="task4"
    DependsOn="task1" "task2" "task3" "task4"
    DependsOn="task1 task2 task3 task4"

    Regards,
    Chris  

    Thursday, February 4, 2010 11:08 AM
  • Hi Chris,

    Sorry for the confusion over the job add /depend command syntax. You can specify tasknames using a comma-delimited list. The following is an example:

    job add 123 /depend:task1name,task2name notepad.exe

    BTW The reason that specifying /depend:"task1name" "task2name" does not generate an error is that job.exe interprets everything at the end (e.g. "task2name") as being part of the command to be executed.

    Regards,

    Patrick
    Thursday, February 4, 2010 6:35 PM