locked
Adding tasks to running job 10 times slower than one in configuring state. RRS feed

  • Question

  • We are developing a system to do Data Parallelism (image processing) and can not use the parametric features since our process is non-numeric.  Our tasks are added dynamically during the process to a running job (what needs to be done is dependant on what is discovered on the image).

    The tasks are add via a web service so the time to add them is limited to <300 seconds.  

    We have run into a barrier where we need to be able to add ~20,000 tasks to a running job.  

    I wrote up a test app to test SP1 vs. R2 RC1 and get the results below.  It was tested on VMs where R2 RC1 had the DBs off the VM on another physical machine.

     

    [SubmitTask() 5000 total]

    Hpc Ver     Job State     Avg. Time / Task (milliseconds)

    SP1          Running               519  

    SP1          Configuring           44

    R2 RC1       Running               496

    R2 RC1       Configuring           68

     

    [SubmitTasks() 5000 total]

    R2 RC1       Running               20

    R2 RC1       Configuring           64

     

    NOTE: When tested on high end physical machine vs. virtual machine for SP1 tests there was little difference in timings.

     

    1. Adding a single task to a running job takes 500ms or 50ms to a job that is still in the configuring state.  

    2. Moving to SubmitTasks() is possible when R2 is released but 20k tasks will still take 6.7 minutes to submit, way over the web method call limit.

     

    None of the new or old methods are performant enough is there a better way to do this?

    What are the limits to the number of tasks in a job?  We are projecting a single job (process one image) would contain 100,000 to a million tasks by the end.

     

    -Scott

    Friday, July 9, 2010 9:09 PM

Answers

  • Spoke with Colin on the phone and have some solutions.

     

    In the near term we are reducing the 20k-50 task groups into collections that a single task will process serially.

     

    In the longer term we have a couple of other solutions. 

    1. Break out large task groups (20k-50k) into individual jobs that are all setup during the job configuring state, make use of the new AddTasks() interface in Hpc 2008 R2.

    2. Research how we might fit into the SOA model of processing and if that model would work better than using basic tasks.  Our un-manged C++ process would then started from some managed code.

     

    For better performing clean-up of old jobs reduce our basic task command line list of parameters in V3 to use the job level environment variables.

     

    Thank you again for your time Colin.

    -Scott

    Monday, July 26, 2010 9:23 PM

All replies

  • Hi Scott,

    Adding tasks to a running job is slower due to the extra operations that are being simultaneously performed on the job. Job level counters need to be updated per task rather than when the job is submitted. There is also other activity to determine dependencies, potential interactions with tasks starting and finishing.

    It is possible that your application can still use a parametric sweep which as you stated will be much quicker to add to the job. It may be necessary to create a control file that contains the parameters that you would normally pass as parameters in the basic tasks that you are adding.

    I am concerned that you are contemplating 1 million tasks. If you continue with basic tasks or parametric sweep then consider doing multiple steps in each task. There is another programming model supported by HPC called SOA that could be ideal for your application.


    -Colin, Microsoft HPC
    Monday, July 12, 2010 4:48 AM
    Moderator
  • Colin,

    As I stated above our system can not use parametric sweep since the unique data doesn't follow an algorithmic pattern.  Though it is possible to roll up some of the large groups of tasks into one task, processed serially internally, not being able to spread out as wide as possible in the cluster is a processing throughput hit and possibly waste of CPU time if parts of the cluster are free at the time.

    The difference between a running job and a configuring job I understand the latter should be quicker but didn't expect a 10x decrease in speed.  More alarming is that a VM on cheap hardware performed at almost the same speed as a high end physical server backed by a large drive array.

     

    1. If a million tasks is a concern what number of tasks per job is acceptable?

    2. How does moving to an SOA model fix the problem of tasks being added slowly or not being able to add so many tasks to a job?  (our cluster task process is 100% C++)

    3. Currently we are not using task dependencies (ran into a bug in Hpc RTM version that is now fixed in SP1) but would like to in the future, how does the performance degrade when working with multiple dependency groups vs. with a single group?

     

    -Scott

     

     

    Monday, July 12, 2010 5:21 PM
  • Spoke with Colin on the phone and have some solutions.

     

    In the near term we are reducing the 20k-50 task groups into collections that a single task will process serially.

     

    In the longer term we have a couple of other solutions. 

    1. Break out large task groups (20k-50k) into individual jobs that are all setup during the job configuring state, make use of the new AddTasks() interface in Hpc 2008 R2.

    2. Research how we might fit into the SOA model of processing and if that model would work better than using basic tasks.  Our un-manged C++ process would then started from some managed code.

     

    For better performing clean-up of old jobs reduce our basic task command line list of parameters in V3 to use the job level environment variables.

     

    Thank you again for your time Colin.

    -Scott

    Monday, July 26, 2010 9:23 PM