Sign in
Microsoft.com
 
Microsoft
 
 
 

locked
How to ensure unique processes for each HPC task RRS feed

 > 

  • Question

  • Question
    Sign in to vote
    0
    Sign in to vote

    We have a local (not Azure) HPC installation using WCF broker on which we have an HPC job that contains a variable number of tasks (can be 100+). Our system is configured such that we allocate as many tasks to each compute node as it has cores. We have noticed that the HPC compute nodes will tend to re-use the same process (HpcServiceHost32.exe) for any additional tasks within the same job. This causes problems due to some unmanaged memory resources not being released that are beyond our control. Is it at all possible to ensure that each tasks gets a unique process, instead of re-using an existing process used for a previous task?

    Thanks in advance.

    Thursday, July 12, 2012 3:15 PM

All replies

  • Question
    Sign in to vote
    0
    Sign in to vote
    try set your resource allocation to unit, and set serviceRequestPrefetchCount to 0 in your service registration file.
    Wednesday, July 18, 2012 1:45 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Unfortunately setting serviceRequestPrefetchCount does not work. The same HPC process is being used for running multiple jobs and we can see the memory usage growing :(

    I'm not sure what setting "resource allocation to unit" means, maybe you meant setting it to node rather than core? We really don't want that because the whole point of using HPC for us is to be able to run a single job per core.

    Any other ideas?

    Wednesday, August 1, 2012 3:05 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    If you have more tasks (or SOA requests) than actual cores, some process will process multiple requests. Say, your cluster have 24 cores and you are running 96 tasks. On average each core/process will have to handle 4 tasks/requests. That's inevitable.

    Friday, October 19, 2012 2:08 PM
  • Question
    Sign in to vote
    0
    Sign in to vote
    If the memory is the issue instead of sending one job with +100 tasks you may send a few jobs with fewer tasks per job.So as in example above 96 tasks / 24 cores -> 1 job with 96 tasks wil cause HpcServiceHost32.exe to handle 4 tasks on 24 core cluster but 4 jobs with 24 tasks each will cause Hpc ServiceHost32.exe to handle only 1 task ( but will be executed 4 times ,once per each job).

    Daniel Drypczewski

    Tuesday, November 6, 2012 7:40 AM