Tuesday, January 27, 2009 2:41 AMI'm interfacing to a CCS 2003 cluster via Matlab's Parallel/Distributed Computing Toolbox, so I may want to ask Matlab people this question... But I've been noticing that when I start a job, it only seems to be able to process tasks using the number of workers that are idle when the job begins. But the tasks within my jobs tend to finish in an unpredictable order and sometimes quite different amounts of time. So what I'd really like is if every task that finished within one job could somehow notify other running/queued jobs that an additional worker is now free. That event only seems to happen when the very last task in a job finishes, so I can have many idle workers despite having pending jobs and/or tasks. Is there some setting I can utilize to make the scheduler do that? I could make every single task its own job, but that's not organizationally ideal. Thanks in advance for any help.
Tuesday, January 27, 2009 5:01 PMModerator
This isn't possible in CCS 2003. To get that behavior, you can submit each task as it's own job (as long as you don't have dependencies between them this should work.
HPC Server 2008 includes a feature called Grow and Shrink Scheduling which does exactly what you are asking for. So you may want to consider an upgrade :-)
- Marked As Answer by Josh BarnardModerator Tuesday, January 27, 2009 5:01 PM
Tuesday, January 27, 2009 5:24 PMThanks for the informative reply. We're planning on upgrading the cluster to HPC 2008 in the near future, now I've got an additional motivation. I'll have to keep an eye out for that feature and make sure my Matlab-submitted jobs can take advantage of it.