giovedì 19 gennaio 2012 16:30
Occasionally we have failures that require us to requeue tasks. That works fine, except we're concerned about cases where the grid is heavily used, and that requeued tasks will be sent to the bottom of the "queued" list and cause this run's results to be delayed behind other runs. Is this a legitimate concern, or will the initial job reserve 1 or more cores to ensure that any requeued tasks immediately run (or worst-case, when the rest of the tasks for this initial job have completed)? I guess the point is, we want to avoid a situation where tasks are requeued but the initial job is forced to wait for hours because someone else snuck in a huge job in the mean-time. Since there should always be one core (or whatever the "minimum" is set to) reserved for use by the job, it seems to me that perhaps any requeued tasks will simply get re-run at the end (at worst-case, if other cores are busy). If there are expected to be such errors during a run (and requeues), would raising the minimum number of cores for the job mitigate the problem? Is there a reason for concern?
If the job has completed and there are still failed tasks, the entire job can be re-submitted at a higher priority. I think this is somewhat of a special case, but requeuing the entire job at a higher priority may address the problem for the case where your last remanining task(s) fail and the job has ended and thus needs to be re-submitted. In this case, even if re-submitting at a higher priority, a large job may have gotten submitted inbetween the time that this job had started, and no cores may be available when re-submitted until a currently-running task has ended. I don't think there's really any way around that aside from allowing currently-running tasks to be pre-empted, but I'd rather not do that.
This thread mentioned a similar question, but it appears that you cannot reset most properties, including priority, when the job is in a running state. Here, the job is in a queued state. In my case, it'll either be running or failed (completed).
Tutte le risposte
martedì 24 gennaio 2012 12:56
the way to achieve this is to cancel the job (unless it's already in failed state), set it to a high priority, and requeue it. The scheduler won't allow a requeued job to go to the head of the waiting list unless it's of high priority. It's designed this way so that nobody can unfairly occupy the cluster for a long time by requeuing a canceled job for multiple times.
You don't have to wait the large job to finish to release resource though. By enabling graceful preemption, the scheduler will release cores to high priority jobs at task bounday (when a task in low priority job finishes). See http://technet.microsoft.com/en-us/library/dd197402(WS.10).aspx
- Contrassegnato come risposta Gary Wolfe lunedì 30 gennaio 2012 15:44
lunedì 30 gennaio 2012 17:55
Thanks for the reply.
I think the key concern was that while a job is running, what happens when you requeue a task? After a good amount of testing (where I'm pretty sure we have the default graceful preemption) I find that the job does not relenquish cores (at least if there are not higher-priority tasks waiting), and requeued tasks are processed as soon as a core is available. As the job finishes, cores are relenquished to other jobs; it seems to me that it's working as desired, and I'm happy with that. It may be that with graceful preemption that a core will not be instantly available where it may have been if preemption were turned off. So, there still may be a wait until the requeued task runs, but at least it will not requeue behind other queued jobs' tasks.
Once the job has ended (failed), it's a different situation. To re-run job failures, you must re-queue. At that point, you can change the priority. If you don't change the priority, you get stuck behind all other previously-queued jobs.
But the bottom-line is, I don't think there's a need to force a cancel on the entire job as long as it's still running -- simply requeueing a task seems to work fine (as long as the job is in a running state which it won't be if the last task fails).
A related question: is it better to change the priority before calling .ConfigureJob() or after? (If in a Failed state, it doesn't seem to matter, but there may be a case that I'm not testing where it might matter.)