none
Why use many tasks in one job? Why not just many single-task jobs? RRS feed

  • Question

  • What is the benefit of having many tasks within one job?

    I am using command line tasks and are not using task dependencies, nor parametric sweep.

    If someone wants to submit multiple tasks in one go, is there any other benefit to putting them as separate tasks all in the same job? Or should I just submit them all as separate jobs with one task each?

    e.g. user wants to submit 1000 tasks and be notified as each completes. If I submit these as 1000 tasks in one job, I can't see what benefits the job itself gives me vs submitting 1000 jobs - other than providing a grouping of tasks that were submitted together. Does it reduce overhead or give better performance in terms of workload on the head node?

    Thanks

    Friday, May 29, 2015 4:18 PM

Answers

  • Putting similar tasks in one job will help:

    1. Reduce the overhead of job. Scheduler allocate resource to jobs, within one job, the system won't log on and log out the user in the compute node as task start and ends

    2. Better management experience, admin can modify the job at once (Priority of your job)

    3. And yes, it reduce the scheduler workloads


    Qiufang Shi

    • Marked as answer by TimJRoberts1 Monday, June 1, 2015 7:47 AM
    Monday, June 1, 2015 12:43 AM

All replies

  • Putting similar tasks in one job will help:

    1. Reduce the overhead of job. Scheduler allocate resource to jobs, within one job, the system won't log on and log out the user in the compute node as task start and ends

    2. Better management experience, admin can modify the job at once (Priority of your job)

    3. And yes, it reduce the scheduler workloads


    Qiufang Shi

    • Marked as answer by TimJRoberts1 Monday, June 1, 2015 7:47 AM
    Monday, June 1, 2015 12:43 AM
  • Thanks! Helps to have something concrete when explaining this to others.
    Monday, June 1, 2015 9:35 AM