How do you determine which mahcines have been allocated for your job?
- Under systems, like PBS, TORQUE, LSF, etc. When a job is submitted and goes into the "Running" state the Workload Manager sets an environment variable such as PBS_NODEFILE or LSB_MCPU_HOSTS, which contains the names of the nodes that have been allocated for the job. Is there a MS Job Scheduler equivalent? Basically, I've created an application that uses the command-line to create a job and then submit it. I use a jobfile as a "template" and then modify the template based on a users input. The problem is that my application starts a client application that then invokes a call to mpiexec. [NOTE: Here template refers to my application template, note the job scheduler template used to define job policy.] My goal is to make sure that the mpiexec uses the resources allocated for the job. The basic pattern looks as follows:query user for job detailsgenerate XML job file from a generic xml job file.create new job:job new /jobfile:[my_jobfile.xml]submit job:job submit /id:[my_job_id]The CommandLine attribute in the Task element of of my job file starts a an application, which in turn starts an MPI application via mpiexec.I would like to be able to make sure mpiexec uses the resources allocated to the job.I know that this is somewhat convoluted, but one of my goals is to create a framework that is generic as possible and allows me ot run under multiple platforms and multiple workload managers. My current abstraction gives me that flexibility so I am hesitant to move to a new implementation unless I can be convinced that it is really the right thing to do.Thanks for any suggestions.Aquil
Answers
There are actually a couple of answers to your question :-) The most direct one is that you can check the CCP_NODES variable for a list of nodes assigned to your job (and how many cores on each of them you've been assigned). For a quick look at some of the other environment variables the scheduler sets to help you out, just submit a job that does "set ccp" and take a look at the output.
The indirect answer to your question is that you don't really have to do anything! The HPC Pack includes MS-MPI, our slightly adapted version of the MPICH2 MPI stack. One of the modifications that we've made to MPICH is to have MPIEXEC pick up lots of stuff from the environment. So doing "job submit /numcores:8-16 mpiexec myapp.exe" will start your application on between 8 and 16 cores, with a rank on each core assigned to your job. You don't need to pass any options into MPIEXEC; it will pick up the necessary information from the command line.
There are a few blog posts we've done that provide additional details on the many options available to you:
http://blogs.technet.com/windowshpc/archive/2006/10/03/fun-and-interesting-ways-to-run-mpi-jobs-on-ccs.aspx
http://blogs.technet.com/windowshpc/archive/2008/09/16/mpi-process-placement-with-windows-hpc-server-2008.aspx
http://blogs.technet.com/windowshpc/archive/2006/06/01/running-mpi-ranks-processes-on-specific-nodes.aspx
Enjoy!
Josh
-Josh- Marked As Answer byJosh BarnardMSFT, OwnerFriday, September 18, 2009 11:41 PM
- Proposed As Answer byJosh BarnardMSFT, OwnerMonday, August 31, 2009 8:13 PM
All Replies
There are actually a couple of answers to your question :-) The most direct one is that you can check the CCP_NODES variable for a list of nodes assigned to your job (and how many cores on each of them you've been assigned). For a quick look at some of the other environment variables the scheduler sets to help you out, just submit a job that does "set ccp" and take a look at the output.
The indirect answer to your question is that you don't really have to do anything! The HPC Pack includes MS-MPI, our slightly adapted version of the MPICH2 MPI stack. One of the modifications that we've made to MPICH is to have MPIEXEC pick up lots of stuff from the environment. So doing "job submit /numcores:8-16 mpiexec myapp.exe" will start your application on between 8 and 16 cores, with a rank on each core assigned to your job. You don't need to pass any options into MPIEXEC; it will pick up the necessary information from the command line.
There are a few blog posts we've done that provide additional details on the many options available to you:
http://blogs.technet.com/windowshpc/archive/2006/10/03/fun-and-interesting-ways-to-run-mpi-jobs-on-ccs.aspx
http://blogs.technet.com/windowshpc/archive/2008/09/16/mpi-process-placement-with-windows-hpc-server-2008.aspx
http://blogs.technet.com/windowshpc/archive/2006/06/01/running-mpi-ranks-processes-on-specific-nodes.aspx
Enjoy!
Josh
-Josh- Marked As Answer byJosh BarnardMSFT, OwnerFriday, September 18, 2009 11:41 PM
- Proposed As Answer byJosh BarnardMSFT, OwnerMonday, August 31, 2009 8:13 PM

