none
Access "machinefile" of the HPC Scheduler RRS feed

  • Question

  • Hi,

    I'm trying to figure out how a software using not MS-MPI but some different MPI implementation can be started with the HPC scheduler. The startup mechanism of the software in MPI mode is as follows: 

    1. The main application is started on one node.

    2. The application reads a machinefile containing all nodes reserved for the job.

    3. The main application uses its own mpiexec to start child processes on the cluster nodes. Those child processes do the actual work then. The application is not compiled against MS-MPI and needs to use it's own special MPI implementation.

    I've managed to start the application under the control of several job queuing systems (PBS, LSF, SGE). Each of those system creates a machinefile in a defined format which can then be accessed and interpreted by the application. However, in case of the Microsoft HPC scheduler it is not clear to me if such a "machinefile" is created by the scheduler and what format this file has. Can someone enlighten me in this regard?

    Best regards

    Felix

    Monday, May 10, 2010 4:11 PM

Answers

  • Hi Felix,

    The HPC Job Scheduler stores all machine information in the CCP_NODES environment variable.  The environment variable contains the information about all nodes allocated to your task, even though the task itself is started on only one node.  The task can use the information in this environment variable to start other processes on other nodes, as needed.  This is exactly the information used by the MS-MPI implementation of mpiexec to start individual MPI processes.

    The CCP_Nodes environment variable has the following format.

    <Number_Of_Nodes> <Machine_Name> <Number_Of_Cores> [<Machine_Name> <Number_Of_Cores> ...]

    The first entry is an integer containing the number of allocated nodes, and the subsequent entries for every node give the machine name and the number of cores allocated on that node.  To start a custom implementation of mpiexec, submit a job with a task requiring the desired number of resources, and then use the information from the CCP_Nodes enviroment variable to launch mpiexec.  For instance, your task could read the info from CCP_Nodes, create a machine file using this info, and then launch the custom mpiexec.

    Best regards,
    Leonid.

     

    Tuesday, May 18, 2010 9:01 PM

All replies

  • Hi Felix,

    The HPC Job Scheduler stores all machine information in the CCP_NODES environment variable.  The environment variable contains the information about all nodes allocated to your task, even though the task itself is started on only one node.  The task can use the information in this environment variable to start other processes on other nodes, as needed.  This is exactly the information used by the MS-MPI implementation of mpiexec to start individual MPI processes.

    The CCP_Nodes environment variable has the following format.

    <Number_Of_Nodes> <Machine_Name> <Number_Of_Cores> [<Machine_Name> <Number_Of_Cores> ...]

    The first entry is an integer containing the number of allocated nodes, and the subsequent entries for every node give the machine name and the number of cores allocated on that node.  To start a custom implementation of mpiexec, submit a job with a task requiring the desired number of resources, and then use the information from the CCP_Nodes enviroment variable to launch mpiexec.  For instance, your task could read the info from CCP_Nodes, create a machine file using this info, and then launch the custom mpiexec.

    Best regards,
    Leonid.

     

    Tuesday, May 18, 2010 9:01 PM
  • Hi Leonid,

    thank you very much for this information. This was exactly the missing piece I was looking for!

    Best regards

    Felix

    Friday, May 21, 2010 8:24 AM