none
workstations sync automatically instead of clusrun - possible? RRS feed

  • Question

  • I am not using Azure (hence hpcsync is not an option).

    I want to be able to put my DLLs/files in a directory so that the workstation nodes pull them automatically for a "local deployment". Is this possible?

    I could run a clusrun but all of my workstation nodes may not be available when I run clusrun

    I am on 2012 R2 with HPC pack Update 3

    So is there a best way to sync/update workstation nodes with SOA dlls/exes in a 'local deployment' scenario?


    • Edited by SRIRAM R Wednesday, June 22, 2016 1:32 PM
    Wednesday, June 22, 2016 1:32 PM

Answers

  • Hi SRIRAM R,

    You may create a test job and specify 'set' or 'set ccp' as the command line for a simple or preparation or release task, then submit the job. After the job finishes, you may check the task output for all the available enviroment variables or those starts with 'ccp'. Below is an example list,

    CCP_TASKID=2
    CCP_TASKCONTEXT=10.143
    CCP_CLUSTER_NAME=HPC-3232516
    CCP_JOBID=10
    CCP_DATA=C:\Program Files\Microsoft HPC Pack 2012\Data\
    CCP_NODES_CORES=1 HPC-3232516 1
    CCP_SCHEDULER=HPC-3232516
    CCP_HOME=C:\Program Files\Microsoft HPC Pack 2012\
    CCP_JOBTEMPLATE=Default
    CCP_MPI_WORKDIR=%USERPROFILE%
    CCP_NUMCPUS=1
    CCP_RERUNNABLE=False
    CCP_LOGROOT_USR=%LOCALAPPDATA%\Microsoft\Hpc\LogFiles\
    CCP_TASKINSTANCEID=1
    CCP_PREVIOUS_JOB_ID=9
    CCP_NODES=1 HPC-3232516 1
    CCP_SERVICEREGISTRATION_PATH=\\HPC-3232516\HpcServiceRegistration
    CCP_RETRY_COUNT=0
    CCP_TASKSYSTEMID=143
    CCP_EXCLUSIVE=False
    CCP_JOBTYPE=Batch
    CCP_LOGROOT_SYS=C:\Program Files\Microsoft HPC Pack 2012\Data\LogFiles\
    CCP_ENVLIST=HPC_PARAMETRIC_INCREMENT,CCP_TASKSYSTEMID,HPC_RUNTIMESHARE,CCP_TASKINSTANCEID,CCP_TASKID,CCP_JOBID,HPC_PARAMETRIC_START,HPC_PARAMETRIC_INDEX,HPC_PARAMETRIC_END
    CCP_RUNTIME=2147483647
    CCP_MPI_NETMASK=10.156.60.0/255.255.252.0
    CCP_COREIDS=0
    ...

    Regards,

    Yutong Sun

    • Marked as answer by SRIRAM R Thursday, June 30, 2016 12:47 PM
    Thursday, June 30, 2016 7:33 AM
    Moderator
  • Hi SRIRAM R,

    This is as expected. The node preparation and release tasks would be in Qeueud state if the job is SOA service job, even when all the sub-tasks are Finished or the service job is Canceled. This is because for the service tasks, scheduler may start new service task instances when the job grows resource on additional nodes or the Canceled job is requeued, then the node preparation and release tasks should also start new sub-tasks to run on the nodes. 

    Regards,

    Yutong Sun


    Monday, July 4, 2016 7:43 AM
    Moderator

All replies

  • Please check this https://technet.microsoft.com/en-US/library/ff919369.aspx , thus one solution for you is to use central deployment

    Option 2: Deploy through Node Preparation Task, this can be specified in the service configuration file (Every time a node is being allocated to the SOA job, this task will be executed first): Modify the SOA service registration file to add the prepareNodeCommandLine (or releaseNodeCommandLine) attributes to the service section.


    Qiufang Shi

    Thursday, June 23, 2016 3:03 AM
  • Any relevant documentation (with examples?)

    Can I call a batch file? If so, where does this file have to reside? what is the syntax to calling this batch (.bat) file in prepareNodeCommandLine attribute?

    Thursday, June 23, 2016 10:13 PM
  • Hi SRIRAM,

      This isn't been public documented as far I as I see. The usage is the same as you add a node preparation task and node release task in a job but the command is executed by the HpcServiceHost32.exe. We will try to add this in the EchoService in our next update. Here is a simple sample I added in the CcpEchoSvc.config:

     <microsoft.Hpc.Session.ServiceRegistration>
        <service assembly="%CCP_HOME%bin\EchoSvcLib.dll"
            prepareNodeCommandLine="whoami"
            releaseNodeCommandLine="cd"
                 contract="EchoSvcLib.IEchoSvc"
                 type="EchoSvcLib.EchoSvc"
                 includeExceptionDetailInFaults="true"
                 maxConcurrentCalls="0"
                 maxMessageSize="655360"
                 serviceInitializationTimeout="60000"
                 enableMessageLevelPreemption="true"
                 soaDiagTraceLevel="Off">
          <!--Below is a sample for adding environment variables to the service-->
          <environmentVariables>
            <add name="myname1" value="myvalue1"/>
            <add name="myname2" value="myvalue2"/>
          </environmentVariables>
        </service>

    Hope this works for you.


    Qiufang Shi

    Friday, June 24, 2016 2:24 AM
  • Thanks.

    So "whoami" in your example could be a be whoami.bat and it has to be in the same folder as the SOA dll on the workstation nodes or does it live in the same folder as the service configuration file on the head node?

    Friday, June 24, 2016 2:39 AM
  • Hi SRIRAM R,

    Yes, you may call a batch file. The prepareNodeCommandLine/prepareNodeCommandLine is invoked by HpcServiceHost32.exe under the working directory where the service dll is located. In the CcpEchoSvc example, the working directory is %CCP_HOME%Bin. So you may place the batch script under the working directory, or it is also feasible to put the batch script on a file server share where the SOA service job runas user has the read access and then use the UNC path of the script in the prepareNodeCommandLine/prepareNodeCommandLine.

    Besides, just echo Qiufang's suggestion for central service deployment, if the service dll is not large in size, central service deployment would be a better choice with regard to the service sync, upgrade and maintenance.

    Regards,

    Yutong Sun

    Friday, June 24, 2016 3:16 AM
    Moderator
  • Thanks - Now, How can prepare/release node tasks get information about the session like  SessionID/JobID?
    Wednesday, June 29, 2016 8:17 PM
  • This information is in the environment variable and usually start with "CCP_". You can start with a node prepare command "set" to get the full list of environment variables.


    Qiufang Shi

    Thursday, June 30, 2016 1:16 AM
  • Thanks. Where do I run the 'set' command ? in PowerShell on headnode? Any example of how to get the full list of env. variables and what they mean?

    (Found an old page on 2008 R2 - https://technet.microsoft.com/en-us/library/gg286970(v=ws.10).aspx )

    Still curious in knowing the full and update list of env. variables.


    Thursday, June 30, 2016 3:09 AM
  • Hi SRIRAM R,

    You may create a test job and specify 'set' or 'set ccp' as the command line for a simple or preparation or release task, then submit the job. After the job finishes, you may check the task output for all the available enviroment variables or those starts with 'ccp'. Below is an example list,

    CCP_TASKID=2
    CCP_TASKCONTEXT=10.143
    CCP_CLUSTER_NAME=HPC-3232516
    CCP_JOBID=10
    CCP_DATA=C:\Program Files\Microsoft HPC Pack 2012\Data\
    CCP_NODES_CORES=1 HPC-3232516 1
    CCP_SCHEDULER=HPC-3232516
    CCP_HOME=C:\Program Files\Microsoft HPC Pack 2012\
    CCP_JOBTEMPLATE=Default
    CCP_MPI_WORKDIR=%USERPROFILE%
    CCP_NUMCPUS=1
    CCP_RERUNNABLE=False
    CCP_LOGROOT_USR=%LOCALAPPDATA%\Microsoft\Hpc\LogFiles\
    CCP_TASKINSTANCEID=1
    CCP_PREVIOUS_JOB_ID=9
    CCP_NODES=1 HPC-3232516 1
    CCP_SERVICEREGISTRATION_PATH=\\HPC-3232516\HpcServiceRegistration
    CCP_RETRY_COUNT=0
    CCP_TASKSYSTEMID=143
    CCP_EXCLUSIVE=False
    CCP_JOBTYPE=Batch
    CCP_LOGROOT_SYS=C:\Program Files\Microsoft HPC Pack 2012\Data\LogFiles\
    CCP_ENVLIST=HPC_PARAMETRIC_INCREMENT,CCP_TASKSYSTEMID,HPC_RUNTIMESHARE,CCP_TASKINSTANCEID,CCP_TASKID,CCP_JOBID,HPC_PARAMETRIC_START,HPC_PARAMETRIC_INDEX,HPC_PARAMETRIC_END
    CCP_RUNTIME=2147483647
    CCP_MPI_NETMASK=10.156.60.0/255.255.252.0
    CCP_COREIDS=0
    ...

    Regards,

    Yutong Sun

    • Marked as answer by SRIRAM R Thursday, June 30, 2016 12:47 PM
    Thursday, June 30, 2016 7:33 AM
    Moderator
  • Have a weird issue with PrepareNodeCommandLine. the task is being queued and the 'main' tasks are getting executed.. why?

    my preparenodecommandline copies a set of files for the soa dll to process and its failing. when i looked at job details, i found out that preparenodecommandline task was queued..

    Here is the screenshot:

    https://1drv.ms/i/s!AnOkuFPvV8aAivImEtXFVye_C_3SwQ

    Monday, July 4, 2016 5:40 AM
  • Hi SRIRAM R,

    This is as expected. The node preparation and release tasks would be in Qeueud state if the job is SOA service job, even when all the sub-tasks are Finished or the service job is Canceled. This is because for the service tasks, scheduler may start new service task instances when the job grows resource on additional nodes or the Canceled job is requeued, then the node preparation and release tasks should also start new sub-tasks to run on the nodes. 

    Regards,

    Yutong Sun


    Monday, July 4, 2016 7:43 AM
    Moderator
  • Thank you. I will dig further into my preparenodetask logic then.
    Monday, July 4, 2016 2:53 PM