none
Submitting Jobs through Job Files/Job Submission Scripts/JSDL

    Kérdés

  • I've been searching high and low...mostly low for an example of submitting jobs to the Windows HPC Server Job Scheduler via a submission script.  With PBS, LSF, SGE, ... you can create job submission scripts that are similar to shell scripts to describe the resources needed for a job as well as the environment needed to run the job and the application to run.

    All of the documentaiton that I've seen so far has mentioned a "job file", but I've had a hard time finding one.  I did see a niftty little webcast by Steven Newhouse about using JSDL, but that seems a bit heavyweight for what I want to do.  So here are my questions:

    1. Is there a way to perform a jobs submission from the command-line, where I could use the syntax: --> job submit some_file.xml?
    2. If the answer to question #1 is yes, then where can I find examples of the xml description needed to submit the job?
    3. If the answer to #1 is no, then what suggestions would you make for users who have applications that need to submit batch jobs, and basically feed the results
    of the stdout, stderr from the job submission to another application to parse the results?

    I know there are lots of cool APIs for doing this via the .NET framework, but I am working up to that. ;)

    Regards,

    Aquil H. Abdullah
    2009. augusztus 24. 21:26

Válaszok

  • So after a little bit more searching I was able to work out a couple of ideas:

    1. You can create .bat files that configure and run your jobs... (.bat the .sh equivalent). While this idea doesn't thrill me it does get me most of what I need.
    2. You can create a simple job and save it and use it as a template. For example, you can request a job that just performs a dir, save it, and then submit a job with the command:
      job /jobfile my_job.xml.

    NOTE: The one caveat to this is that you will get the same jobid when the job is submitted so if you want  a new jobid you have to submit the job using the new argument...

    Here is an example:

    Start the HPC Job Manager: Go to Actions -> Job Submission -> New Job. [Describe the job as you see fit]
    Then select save as...

    This generates an XML file with the basic skeleton:

    <Job>
      <Dependencies/>
      <Tasks>
         <Task/>
       </Tasks>
    </Job>

    If you like you can use this file as a *Template* for other jobs that you would like to submit from the command-line. And you can submit them as follows:

    C:\Users\aabdullah>job new /jobname:test /jobfile:job_test.xml

    Created job, ID: 426

     

    Step 2:

    C:\Users\aabdullah\Documents>job submit /Id:426


    *NOTE: The use of the word template here is NOT the same as a Job Template which is used to set policies for a job.

    One of the things that I plan on doing is writing an application that performs a search and replace on the Job attributes that describe the resources and the Task attributes that describe that application that I want to run and then submits the job.

    CONCLUSION: The Microsoft provides a plethora of ways to interface with the HPC Job Scheduler and eventually I'll start writing applications that use the .NET Micrsoft.Hpc.Scheduler namespace, but until then I can use an XML job file to submit jobs.  The best part is that I don't have to change my existing application, because I can parse stdout and stderr.

    Complete job file:

    <?xml version="1.0" encoding="utf-8"?>
    <Job Version="2.000" 
         Id="443" 
         Name="Test" 
         SubmitTime="8/26/2009 2:36:57 PM" 
         CreateTime="8/26/2009 2:32:58 PM" 
         ChangeTime="8/26/2009 2:32:58 PM" 
         UnitType="Core" 
         MinCores="1" 
         MaxCores="1" 
         RuntimeSeconds="600" 
         RunUntilCanceled="true" 
         IsExclusive="true" 
         ErrorCode="0" 
         ErrorParams="" 
         State="Configuring" 
         PreviousState="Failed" 
         UserName="MyDomain\aabdullah" 
         JobType="Batch" 
         Priority="Normal" 
         IsBackfill="false" 
         NextTaskNiceID="2" 
         HasGrown="false" 
         HasShrunk="false" 
         OrderBy="" 
         RequestCancel="None" 
         RequeueCount="1" 
         AutoRequeueCount="0" 
         FailureReason="None" 
         PendingReason="None" 
         AutoCalculateMax="false" 
         AutoCalculateMin="false" 
         ParentJobId="0" 
         ChildJobId="0" 
         NumberOfCalls="0" 
         NumberOfOutstandingCalls="0" 
         CallDuration="0" 
         CallsPerSecond="0" 
         ProjectId="3" 
         JobTemplateId="1" 
         OwnerId="9" 
         ClientSourceId="6" 
         Project="My Test" 
         JobTemplate="Default" 
         DefaultTaskGroupId="443" 
         Owner="MyDomain\aabdullah" 
         ClientSource="HpcJobManager" 
         xmlns="http://schemas.microsoft.com/HPCS2008/scheduler/">
        <Dependencies />
        <Tasks>
            <Task Version="2.000" 
         Id="1131" 
         SubmitTime="8/26/2009 2:36:57 PM" 
         CreateTime="8/26/2009 2:35:52 PM" 
         EndTime="8/26/2009 2:36:57 PM" 
         ChangeTime="8/26/2009 2:36:57 PM" 
         ErrorCode="0" 
         ErrorParams="" 
         State="Configuring" 
         PreviousState="Failed" 
         ParentJobId="443" 
         RequestCancel="None" 
         Closed="false" 
         RequeueCount="1" 
         AutoRequeueCount="0" 
         FailureReason="None" 
         PendingReason="None" 
         InstanceId="0" 
         Output="" 
         RecordId="1131" 
         Name="MyTask" 
         MinCores="1" 
         MaxCores="1" 
         MinSockets="1" 
         MaxSockets="1" 
         MinNodes="1" 
         MaxNodes="1" 
         NiceId="1" 
         CommandLine="dir" 
         WorkDirectory="C:\Users\aabdullah" 
         StdOutFilePath="test_stdout.txt" 
         StdErrFilePath="test_stderr.txt" 
         HasCustomProps="false" 
         IsParametric="false" 
         GroupId="443" 
         ParentJobState="Configuring" 
         UnitType="Core" 
         ParametricRunningCount="0" 
         ParametricCanceledCount="0" 
         ParametricFailedCount="0" 
         ParametricQueuedCount="0" />
        </Tasks>
    </Job>
    2009. augusztus 26. 15:14

Az összes válasz

  • So after a little bit more searching I was able to work out a couple of ideas:

    1. You can create .bat files that configure and run your jobs... (.bat the .sh equivalent). While this idea doesn't thrill me it does get me most of what I need.
    2. You can create a simple job and save it and use it as a template. For example, you can request a job that just performs a dir, save it, and then submit a job with the command:
      job /jobfile my_job.xml.

    NOTE: The one caveat to this is that you will get the same jobid when the job is submitted so if you want  a new jobid you have to submit the job using the new argument...

    Here is an example:

    Start the HPC Job Manager: Go to Actions -> Job Submission -> New Job. [Describe the job as you see fit]
    Then select save as...

    This generates an XML file with the basic skeleton:

    <Job>
      <Dependencies/>
      <Tasks>
         <Task/>
       </Tasks>
    </Job>

    If you like you can use this file as a *Template* for other jobs that you would like to submit from the command-line. And you can submit them as follows:

    C:\Users\aabdullah>job new /jobname:test /jobfile:job_test.xml

    Created job, ID: 426

     

    Step 2:

    C:\Users\aabdullah\Documents>job submit /Id:426


    *NOTE: The use of the word template here is NOT the same as a Job Template which is used to set policies for a job.

    One of the things that I plan on doing is writing an application that performs a search and replace on the Job attributes that describe the resources and the Task attributes that describe that application that I want to run and then submits the job.

    CONCLUSION: The Microsoft provides a plethora of ways to interface with the HPC Job Scheduler and eventually I'll start writing applications that use the .NET Micrsoft.Hpc.Scheduler namespace, but until then I can use an XML job file to submit jobs.  The best part is that I don't have to change my existing application, because I can parse stdout and stderr.

    Complete job file:

    <?xml version="1.0" encoding="utf-8"?>
    <Job Version="2.000" 
         Id="443" 
         Name="Test" 
         SubmitTime="8/26/2009 2:36:57 PM" 
         CreateTime="8/26/2009 2:32:58 PM" 
         ChangeTime="8/26/2009 2:32:58 PM" 
         UnitType="Core" 
         MinCores="1" 
         MaxCores="1" 
         RuntimeSeconds="600" 
         RunUntilCanceled="true" 
         IsExclusive="true" 
         ErrorCode="0" 
         ErrorParams="" 
         State="Configuring" 
         PreviousState="Failed" 
         UserName="MyDomain\aabdullah" 
         JobType="Batch" 
         Priority="Normal" 
         IsBackfill="false" 
         NextTaskNiceID="2" 
         HasGrown="false" 
         HasShrunk="false" 
         OrderBy="" 
         RequestCancel="None" 
         RequeueCount="1" 
         AutoRequeueCount="0" 
         FailureReason="None" 
         PendingReason="None" 
         AutoCalculateMax="false" 
         AutoCalculateMin="false" 
         ParentJobId="0" 
         ChildJobId="0" 
         NumberOfCalls="0" 
         NumberOfOutstandingCalls="0" 
         CallDuration="0" 
         CallsPerSecond="0" 
         ProjectId="3" 
         JobTemplateId="1" 
         OwnerId="9" 
         ClientSourceId="6" 
         Project="My Test" 
         JobTemplate="Default" 
         DefaultTaskGroupId="443" 
         Owner="MyDomain\aabdullah" 
         ClientSource="HpcJobManager" 
         xmlns="http://schemas.microsoft.com/HPCS2008/scheduler/">
        <Dependencies />
        <Tasks>
            <Task Version="2.000" 
         Id="1131" 
         SubmitTime="8/26/2009 2:36:57 PM" 
         CreateTime="8/26/2009 2:35:52 PM" 
         EndTime="8/26/2009 2:36:57 PM" 
         ChangeTime="8/26/2009 2:36:57 PM" 
         ErrorCode="0" 
         ErrorParams="" 
         State="Configuring" 
         PreviousState="Failed" 
         ParentJobId="443" 
         RequestCancel="None" 
         Closed="false" 
         RequeueCount="1" 
         AutoRequeueCount="0" 
         FailureReason="None" 
         PendingReason="None" 
         InstanceId="0" 
         Output="" 
         RecordId="1131" 
         Name="MyTask" 
         MinCores="1" 
         MaxCores="1" 
         MinSockets="1" 
         MaxSockets="1" 
         MinNodes="1" 
         MaxNodes="1" 
         NiceId="1" 
         CommandLine="dir" 
         WorkDirectory="C:\Users\aabdullah" 
         StdOutFilePath="test_stdout.txt" 
         StdErrFilePath="test_stderr.txt" 
         HasCustomProps="false" 
         IsParametric="false" 
         GroupId="443" 
         ParentJobState="Configuring" 
         UnitType="Core" 
         ParametricRunningCount="0" 
         ParametricCanceledCount="0" 
         ParametricFailedCount="0" 
         ParametricQueuedCount="0" />
        </Tasks>
    </Job>
    2009. augusztus 26. 15:14
  • Hi Aquil
    I guess it's worth noting that another equivalent for a shell type submission would be to make use of PowerShell. As the name suggests this is an extremely powerful way to manipulate jobs and tasks, have a look here for some good tips:
    As you say there are many ways to do this, it's just a case of finding the most appropriate for your needs.
    Cheers
    Dan
    2009. augusztus 27. 14:52