none
Communication between task processes and the job process RRS feed

  • Question

  • Hi,

     

    Being new to HPCS, does it provide a way of establishing some kind of communication link between a job and a task created by it?

    I'm talking about something like stdout , but with an ability to receive streaming output from the task in real-time.

    The basic needs for such communication are:

    1. Being able to monitor if a task is "alive" in real-time (i.e. the process hasn't hanged) [does HPCS provide a built-in way to do this?]

    2. Being able to monitor the progress of a specific task

    3. Being able to send a command to the job process to create a new task (we want the job process to be the only process responsible for creating new tasks).

     

    MPI is no good for our purpose, since for MPI we have to specify all the task processes at once and in our case tasks are created dynamically.

     

    I'd appreciate your help much.

     

    Alex.

     

    Wednesday, June 9, 2010 12:28 PM

Answers

  • Hi Alex,

    Could you share some more details about what you're currently doing and how Windows HPC may help you in the future?

    With better knowledge of your scenario, maybe we can suggest some other approaches for Windows HPC utilization. If you're interested, please send me an email to lutom@microsoft.com

    Thanks,
    Łukasz

    Wednesday, June 30, 2010 4:42 PM

All replies

  • Hi Alex,

    After seeing some of your other messages, I guess you have this figured out, but let me briefly answer your questions, so maybe it can be useful for others as well (some of this applies only to R2 release):

    First of all, there is no job-level process and task-level process. Everything, that is running is a separate task. However, application running as a task within the job can be aware of it and can access job/task properties by using scheduler APIs. To connect to scheduler and obtain job object you need scheduler's hostname and job's ID. These are provided to the task via environment variables: CCP_SCHEDULER, CCP_JOBID.

    Now having a first task of a job as a 'manager', which connects to the scheduler and opens its job object you can:

    For Q1: If the hanging process causes task to fail, you can detect this by subscribing for ISchedulerJob.OnTaskState events.

    For Q2: You can use ISchedulerJob.Progress and ProgressMessage properties to report overall job's progress based on the communication between 'manager' task and other tasks within the job. Progress and message will be visible in JobManager GUI and via CLI/Powershell tools.

    For Q3. You can use ISchedulerJob.SubmitTask(s) to create new tasks within the job.

    I hope this information will be useful. Please let me know if you have any questions.

    Best regards,
    Łukasz

    Friday, June 18, 2010 4:35 PM
  • Hi Alex,

    After seeing some of your other messages, I guess you have this figured out, but let me briefly answer your questions, so maybe it can be useful for others as well (some of this applies only to R2 release):

    First of all, there is no job-level process and task-level process. Everything, that is running is a separate task. However, application running as a task within the job can be aware of it and can access job/task properties by using scheduler APIs. To connect to scheduler and obtain job object you need scheduler's hostname and job's ID. These are provided to the task via environment variables: CCP_SCHEDULER, CCP_JOBID.

    Now having a first task of a job as a 'manager', which connects to the scheduler and opens its job object you can:

    For Q1: If the hanging process causes task to fail, you can detect this by subscribing for ISchedulerJob.OnTaskState events.

    For Q2: You can use ISchedulerJob.Progress and ProgressMessage properties to report overall job's progress based on the communication between 'manager' task and other tasks within the job. Progress and message will be visible in JobManager GUI and via CLI/Powershell tools.

    For Q3. You can use ISchedulerJob.SubmitTask(s) to create new tasks within the job.

    I hope this information will be useful. Please let me know if you have any questions.

    Best regards,
    Łukasz

    Lukaz,

    I appreciate your help, yet I want to clarify a few things.

     

    First of all, tasks in our system vary significantly in their run time. Some tasks may take minutes, while others may take a day. Hence subscribing to ISchedulerJob.OnTaskState events would not help here - there is no constant time interval which would indicate non-responsiveness of a task (we can't estimate how much time it should take for a task to finish).

    Then, I was indeed asking about progress, but, I insist, on the task level. I mean something like ISchedulerJob.Progress and ProgressMessage properties of a task, so progress of each task could be inspected. I understand this feature has not yet been implemented, and that's why I was asking about a communication link between a task in a job and a job (let it be a "manager task"), so each task could report its progress separately to the manager.

    Regarding ISchedulerJob.SubmitTask(s) - yes, this is what we're using now. I was hoping there is a way to eliminate communication overhead and send application-specific requests to the manager (e.g. create 1000 tasks of type X, rather than 1000 requests to create one task of type X). Yet I see that the performance of task submission is reasonable, so this method is acceptable.

     

    In a perfect situation, we'd like a task to be able to modify its custom properties while it's running, and this change would raise an event in the manager task, so it would be able to retrieve the updated properties. I understand this is not implemented, so I'm looking for a workaround. Does it mean we will have to implement our own communication line via TCP/IP between the tasks and the manager task for these needs?

     

    Thank you and I hope I made our requirements clearer.

     

    Thanks again,

    Alex.

     

     

     

    Sunday, June 20, 2010 8:41 AM
  • Hi Alex,

    There is currently no support for the features you require or similar features currently exist, but only on the job level.

    We really appreciate your feedback and your suggestions will be incorporated into our list of customer requests.

    One more question to better understand your case, which is also related with another thread on this forum, where I suggested use of Activation Filter (http://social.microsoft.com/Forums/en-US/windowshpcsched/thread/604273e4-8258-4310-9b29-633bbc3437d4).  What are the main reasons that you want to operate on the task level? Is it about the performance? How about running one master/manager job which will submit other jobs (which will be able to use progress message, activation filter) or running management app out of the job/task infrastructure?

    Best regards,
    Łukasz

    Monday, June 21, 2010 7:23 PM
  • Hi Lucasz,

     

    At first we also had the idea of using jobs as tasks, but then realized jobs lack an important feature that tasks don't: dependency. Mentioning the performance, I also think it's an issue.

     

    Eventually we already have the infrastructure implemented in-house. We were just looking for an option to utilize Windows HPC and get rid of big part of our code which is responsible for job/task management and handling. Indeed Windows HPC seemed to match our home-grown concept very well at first, but looking deeper discovered lack of some important (for us) features that we already have implemented and working. It is not worthy for us to develop too many workarounds to force our concept on HPC. Hence we decided not to take the HPC direction yet and keep checking for updates in the future.

     

    Thank you very much for your help, and I'll be happy to contribute to the customer requests list.

     

    Alex.

    Sunday, June 27, 2010 11:48 AM
  • Hi Alex,

    Could you share some more details about what you're currently doing and how Windows HPC may help you in the future?

    With better knowledge of your scenario, maybe we can suggest some other approaches for Windows HPC utilization. If you're interested, please send me an email to lutom@microsoft.com

    Thanks,
    Łukasz

    Wednesday, June 30, 2010 4:42 PM