Sign in
Microsoft.com
 
Microsoft
 
 
 

locked
Finished tasks in one job not freeing workers for other jobs' pending tasks RRS feed

 > 

  • Question

  • Question
    Sign in to vote
    0
    Sign in to vote
    I'm interfacing to a CCS 2003 cluster via Matlab's Parallel/Distributed Computing Toolbox, so I may want to ask Matlab people this question... But I've been noticing that when I start a job, it only seems to be able to process tasks using the number of workers that are idle when the job begins. But the tasks within my jobs tend to finish in an unpredictable order and sometimes quite different amounts of time. So what I'd really like is if every task that finished within one job could somehow notify other running/queued jobs that an additional worker is now free. That event only seems to happen when the very last task in a job finishes, so I can have many idle workers despite having pending jobs and/or tasks. Is there some setting I can utilize to make the scheduler do that? I could make every single task its own job, but that's not organizationally ideal. Thanks in advance for any help.
    Tuesday, January 27, 2009 2:41 AM

Answers

  • Question
    Sign in to vote
    0
    Sign in to vote

    Tony,

    This isn't possible in CCS 2003.  To get that behavior, you can submit each task as it's own job (as long as you don't have dependencies between them this should work.

    HPC Server 2008 includes a feature called Grow and Shrink Scheduling which does exactly what you are asking for.  So you may want to consider an upgrade :-)

    Thanks,
    Josh

    -Josh
    Tuesday, January 27, 2009 5:01 PM
    Moderator

All replies

  • Question
    Sign in to vote
    0
    Sign in to vote

    Tony,

    This isn't possible in CCS 2003.  To get that behavior, you can submit each task as it's own job (as long as you don't have dependencies between them this should work.

    HPC Server 2008 includes a feature called Grow and Shrink Scheduling which does exactly what you are asking for.  So you may want to consider an upgrade :-)

    Thanks,
    Josh

    -Josh
    Tuesday, January 27, 2009 5:01 PM
    Moderator
  • Question
    Sign in to vote
    0
    Sign in to vote
    Thanks for the informative reply. We're planning on upgrading the cluster to HPC 2008 in the near future, now I've got an additional motivation. I'll have to keep an eye out for that feature and make sure my Matlab-submitted jobs can take advantage of it.
    Tuesday, January 27, 2009 5:24 PM