locked
Set Parametric Sweep task min memory precondition. RRS feed

  • Question

  • Hi all , 

    How can I set parametric task memory precondition ? ( ~1% of my tasks get not enough available memory exception )

    (Something like ISchedulerJob.MinMemory Property but for task )

    Regards, 

    Shai.

     

    Tuesday, May 3, 2011 8:13 AM

Answers

  • Hi Shai,

      Our scheduler system won't aware of the requirement on RAM of the task. That being said, your requirement can't be easily served. Some workarounds:

    1. Make your own logic to requeue your failed job when dectecting failure reason is "not enough mememory" -- Your own monitoring service can serve this needs

    2. From v3sp1, you can submit job from a running task. If you only have one task in your job, you can add another task (depend on the first one). This task does nothing but return 0 if the first task passes, but submit a new same job if it detects the failure of the first task because of "not enough memory". The new job will in the end of the job queue

    Thanks


    Qiufang Shi
    • Marked as answer by Shay_Segev Monday, May 16, 2011 10:56 AM
    Thursday, May 12, 2011 7:46 AM

All replies

  • Hi Shai,

      Can I understand your purpose of getting not enough memory exception? For the scheduler system, it tries to allocate resources to job/tasks and manages/monitoring the execution as well as taking care of the result. From my understanding, your task application shall handle the issue of memory availability and takes the correct action (Either exit with error or retry, or exclude this node from the job...)

    Thanks,

    Qiufang


    Qiufang Shi
    Tuesday, May 10, 2011 1:23 AM
  • Hi , 

    At my cluster each compute node has 24 GB of RAM and 8 cores ,

    Each core can compute different task application that consumes different amount of memory , 

    So if 7 cores computes 7 tasks application that already consume 23 GB of the total ram , and on the queue I have tasks that need more than 1 GB of ram and tasks that need less than 1 GB of ram - I want the scheduler to allocate that core to task that need less than 1 GB of ram .

    My task application can not requeue itself or allocate its own resources according memory availability  .

    Regards , 

    Shai . 

    Wednesday, May 11, 2011 11:03 AM
  • Hi Shai,

      Our scheduler system won't aware of the requirement on RAM of the task. That being said, your requirement can't be easily served. Some workarounds:

    1. Make your own logic to requeue your failed job when dectecting failure reason is "not enough mememory" -- Your own monitoring service can serve this needs

    2. From v3sp1, you can submit job from a running task. If you only have one task in your job, you can add another task (depend on the first one). This task does nothing but return 0 if the first task passes, but submit a new same job if it detects the failure of the first task because of "not enough memory". The new job will in the end of the job queue

    Thanks


    Qiufang Shi
    • Marked as answer by Shay_Segev Monday, May 16, 2011 10:56 AM
    Thursday, May 12, 2011 7:46 AM
  • Thank you for your advice.

    I will try it.

    Shai.

    Monday, May 16, 2011 7:57 AM