none
Creating a Job with multiple tasks each with different resource requirements RRS feed

  • Question

  • Currently it seems like I can use Node Groups and Job Templates to control the problem of different jobs with different resource requirments running on the appropriate compute node.  However, I have jobs with multiple tasks where the different tasks have varying resource requirements.  Is there some way to solve this problem with Node Groups and Job Templates?

    An example:  I have Node Group A and B each with 5 compute nodes in it.  I have a Job that has 2 tasks.  The first task requires a node from Group A and the second task requires a node from Group B.

    The two solutions I have thought of for this problem:

    1. Create 2 different jobs each with one task and make the second job dependent on the first job.  The first job uses a template that requires node group A and the second job uses a template that requires node group B.  However I don't think HPC supports job dependencies since job 2 can only start after job 1 is finished.
    2. Create 1 job with 2 tasks and assign the template at the task level.  However there is no way to use templates (to utilize node groups) at the task level.

    Any ideas?

    Thanks!


    Scott
    • Moved by Alex Sutton Friday, July 16, 2010 12:05 AM (From:Windows HPC Server Developers - General)
    Wednesday, July 14, 2010 9:29 PM

Answers

  • I am sorry but neither job dependencies nor a job with 2 tasks, each with a different resource requirement is supported right now.

    However, there are a couple of ways you could get this working:

    1) You could use the scheduler API provided in the Windows Hpc Server 2008 SDK to submit the jobs. The scheduler API lets you create jobs, configure jobs, submit jobs and monitor them programmatically. You could submit the first job and wait for it to finish before submitting the second one. The scheduler API allows a client to receive events on change in job state for example when a job enters the finished state. You could use the API to write a client that submits the first job, waits for it to finish and then submits the second one. Here are a few helpful links

    MSDN entry for the scheduler interface http://msdn.microsoft.com/en-us/library/microsoft.hpc.scheduler(v=VS.85).aspx

    SDK download at http://www.microsoft.com/downloads/details.aspx?familyid=12887DA1-9410-4A59-B903-693116BFD30E&displaylang=en

    2) Another way to work around this would be to cache your credentials on every compute node in Group A such that you can submit job2 as the last thing that the task in job 1 does. You could use the cluscfg setcreds command on the compute nodes to cache the credentials on those machines. That will allow the task in job 1 to submit job 2 without having to wait for your credentials. However, as you realize this is really a hack and you would have to update the credentials on each compute node, anytime your password changed.

    I would recommend the first workaround using the scheduler API.

     

     

    Thursday, July 22, 2010 12:39 AM
    Moderator

All replies

  • I am sorry but neither job dependencies nor a job with 2 tasks, each with a different resource requirement is supported right now.

    However, there are a couple of ways you could get this working:

    1) You could use the scheduler API provided in the Windows Hpc Server 2008 SDK to submit the jobs. The scheduler API lets you create jobs, configure jobs, submit jobs and monitor them programmatically. You could submit the first job and wait for it to finish before submitting the second one. The scheduler API allows a client to receive events on change in job state for example when a job enters the finished state. You could use the API to write a client that submits the first job, waits for it to finish and then submits the second one. Here are a few helpful links

    MSDN entry for the scheduler interface http://msdn.microsoft.com/en-us/library/microsoft.hpc.scheduler(v=VS.85).aspx

    SDK download at http://www.microsoft.com/downloads/details.aspx?familyid=12887DA1-9410-4A59-B903-693116BFD30E&displaylang=en

    2) Another way to work around this would be to cache your credentials on every compute node in Group A such that you can submit job2 as the last thing that the task in job 1 does. You could use the cluscfg setcreds command on the compute nodes to cache the credentials on those machines. That will allow the task in job 1 to submit job 2 without having to wait for your credentials. However, as you realize this is really a hack and you would have to update the credentials on each compute node, anytime your password changed.

    I would recommend the first workaround using the scheduler API.

     

     

    Thursday, July 22, 2010 12:39 AM
    Moderator
  • Let me ask some follow up questions on this...

    1. How do you call job 2 from job 1 as a task?

    2. Do you need to use the cluscfg setcreds on eash compute node, or just the head node?

    3. Do you use the power shell to run hte cluscfg command, or can you put this in a job as the first task?

    Thursday, March 29, 2012 3:03 PM
  • 1. I assume you are trying to submit job 2 from job 1. In that case you can have the submission of job 2 as the commandline (or part of a commandline) of a task in job 1. This could be done from the Command line interface, powershell or using the scheduler API in the Windows HPC Server 2008 R2 SDK SP3 available at http://www.microsoft.com/download/en/details.aspx?id=28016 (documentation at http://msdn.microsoft.com/en-us/library/cc904930(v=vs.85).aspx). A simple example from the command line is

    c:>job submit  job submit hostname

    This results in job 1 getting submitted with a task that has a commandline "job submit hostname". This task in job 1 runs and submits job 2 with a task that has a command line "hostname"

    2. If you are using Windows Hpc Server 2008 R2 SP1 or later, you can do cluscfg setcreds just once, on the headnode or any client or compute node, for the user that is going to run these jobs. In that case, you do not need to run cluscfg setcreds on every compute node. If you are using a version earlier than Windows Hpc Server 2008 R2 SP1 you need to run cluscfg setcreds on every compute node.

    3. Cluscfg setcreds can be run from the normal commandline. So you do not need to use powershell to run it. Moreover, if you are using Windows Hpc Server 2008 R2 SP1 you need to do this just once for each user.

    Friday, March 30, 2012 4:44 PM
    Moderator