none
Prevent a task to be split among multiple nodes RRS feed

  • Question

  • I have a large job with thousands of tasks, some do need a whole computing node (because they need more memory) some others can share a node.

    I set my larger tasks with 8 cores and the other ones with 4 cores (my nodes have max up to 8 nodes). However I noticed recently that the HPC Scheduler sometimes will run one of the large tasks on 2 nodes and reserve 4 cores on each. In my case this doesn't make any sense because the same task cannot run concurrently on two nodes. Is there any way to customize this behavior?


    Erik Putrycz - Solution Architect - Apption Software


    • Edited by CadErikWork Wednesday, September 26, 2012 3:14 PM
    Wednesday, September 26, 2012 3:14 PM

Answers

  • Hi,

    I had a chat with our team and here are the conlusions

    1. for you you can set /numnodes=1 property at  task level, so your big task will take an entire node with 8 cores all the time

    2. In our next release, we will add new features to properly address the issue, you are welcome to try hpc server 2012 beta right now.

    Michael

    • Marked as answer by CadErikWork Thursday, September 27, 2012 3:13 PM
    Thursday, September 27, 2012 12:21 AM

All replies

  • Hi,

    I had a chat with our team and here are the conlusions

    1. for you you can set /numnodes=1 property at  task level, so your big task will take an entire node with 8 cores all the time

    2. In our next release, we will add new features to properly address the issue, you are welcome to try hpc server 2012 beta right now.

    Michael

    • Marked as answer by CadErikWork Thursday, September 27, 2012 3:13 PM
    Thursday, September 27, 2012 12:21 AM
  • Consider using exclusive=true for the job.  For each task that needs a single node, set exclusive=true for the task as well.  If the issue is genuinely a memory issue for the task, be sure to reduce the number of cores for that task to the number it will actually use.  For tasks that can share a node, leave exclusive=false (which is the default).  Note that, because the job  must be exclusive to allow a task to be declared exclusive, tasks that can share will only be sharing with other tasks within that job.
    Thursday, September 27, 2012 3:02 PM
  • Hi,

    I had a chat with our team and here are the conlusions

    1. for you you can set /numnodes=1 property at  task level, so your big task will take an entire node with 8 cores all the time

    2. In our next release, we will add new features to properly address the issue, you are welcome to try hpc server 2012 beta right now.

    Michael

    I will try setting the number of nodes to 1, I would like to avoid setting the task or job to exclusive - the nodes can be shared among jobs.

    Concerning 2, I wouldn't mind installing HPC 2012 beta on our prod cluster but does the headnode really require windows server 2012?


    Erik Putrycz - Solution Architect - Apption Software

    Thursday, September 27, 2012 3:13 PM
  • Be aware that setting the /numnodes=1 property to a task requires that the job is also using nodes as its UnitType.  This effectively causes all tasks to be exclusive, so they cannot share with other tasks even among those that are within that job.  Using exclusive does mean that they cannot share among other jobs.  Using exclusive for the job but having tasks that are not exclusive is less severe than using node as the unittype of the job.
    Thursday, September 27, 2012 3:20 PM
  • Hi,

    I had a chat with our team and here are the conlusions

    1. for you you can set /numnodes=1 property at  task level, so your big task will take an entire node with 8 cores all the time

    2. In our next release, we will add new features to properly address the issue, you are welcome to try hpc server 2012 beta right now.

    Michael

    I'm running into the exact same problem.

    HPC 2012 is out now. Does it include a solution to this problem? If so, what do I need to do?

    Thursday, May 29, 2014 9:48 PM
  • Hi,

    I had a chat with our team and here are the conlusions

    1. for you you can set /numnodes=1 property at  task level, so your big task will take an entire node with 8 cores all the time

    2. In our next release, we will add new features to properly address the issue, you are welcome to try hpc server 2012 beta right now.

    Michael

    I'm running into the exact same problem.

    HPC 2012 is out now. Does it include a solution to this problem? If so, what do I need to do?

    I recently upgraded our system. It doesn't solve this problem. There is a new feature to run a *job* on a single node but not run each individual task on a single node. The only workaround I believe is to create multiple jobs with unit set to node and filters on number of cores.

    Erik

    Thursday, May 29, 2014 11:38 PM
  • Any plans to fix this problem? We have the same issue too, and all of the solutions cause the grid to be sub-optimal. The scheduler should always allocate X cores on a single node. In our case it makes no sense to allocate X cores across > 1 node.

    Thursday, September 25, 2014 2:34 PM
  • Hi TimJRoberts1, I'd like to understand how do you monitor one task is allocated X cores across nodes? Does it run on more than 1 node simultaneously? Could you provide repro steps?
    Also did you verify whether Clark's reply solve your problem? Thanks.

    Friday, September 26, 2014 10:05 AM
  • Here is a C# Nunit test that reproduces the problem. I ran this on an HPC cluster with 2 compute nodes, each with 4 cores and 1 socket.

    I submit a job with 1 slow task, in order to make an uneven number of nodes available.
    I then submit a job with 4 tasks - each task requires 2 cores. I then check that all of the tasks only executed on one node. This always fails with "Should have only run on 1 core but ran on 2 cores (MyNodeA,MyNodeB)" One of the tasks gets split across 2 nodes.
            [TestFixture]
            public class HpcTests
            {
                [Test]
                public void Should_run_each_task_on_only_one_node()
                {
                    const string hpcServer = "HOSTNAME_OF_HPC_SERVER_GOES_HERE";
                    const string hpcUsername = "USERNAME_GOES_HERE";
                    const string hpcPassword = "PASSWORD_GOES_HERE";
                    
                    const int slowTaskSleepInSeconds = 20;
                
                    const int taskSleepInSeconds = 3;
                    const int numberOfTasks = 4;
                    const int maxTimeToWait = (taskSleepInSeconds * 2) + 3;
                    
                    const int coresPerTask = 2;
                
                    var scheduler = new Scheduler();
                    scheduler.Connect(hpcServer);
                    var jobCompletedResetEvent = new ManualResetEvent(false);
                    ISchedulerJob job = ((IScheduler) scheduler).CreateJob();
                    job.OnJobState += (sender, arg) =>
                    {
                        Console.WriteLine("jobid: " + arg.JobId + ", oldstate: " + arg.PreviousState + ", newstate: " + arg.NewState);
                        if (arg.NewState == JobState.Finished || arg.NewState == JobState.Failed)
                        {
                            jobCompletedResetEvent.Set();
                        }
                    };
                    
                    SubmitJob(scheduler.CreateJob(), scheduler, 1, 1, slowTaskSleepInSeconds, hpcUsername, hpcPassword); // slow task that requires 1 core
                    var stopwatch = Stopwatch.StartNew();
                    var jobId = SubmitJob(job, scheduler, numberOfTasks, coresPerTask, taskSleepInSeconds, hpcUsername, hpcPassword);
                    Assert.IsTrue(jobCompletedResetEvent.WaitOne(TimeSpan.FromSeconds(maxTimeToWait)), "Should have completed " + numberOfTasks + "*" + taskSleepInSeconds + "s tasks requiring " + coresPerTask + " coresPerTask in " + maxTimeToWait + "s");
                    stopwatch.Stop();
                    Console.WriteLine("job completed in " + stopwatch.ElapsedMilliseconds + "ms");
                    var loadedJob = scheduler.OpenJob(jobId);
    
                    Console.WriteLine("jobid: " + jobId + " ran on nodes: " + String.Join(",", loadedJob.AllocatedNodes));
                    foreach (var result in loadedJob.GetTaskList(null, null, false).OfType<ISchedulerTask>())
                    {
                        var allocatedNodes = String.Join(",", result.AllocatedNodes);
                        Console.WriteLine("taskID " + result.TaskId.JobTaskId + " got result: " + result.Output.TrimEnd() + " on allocatednodes: " + allocatedNodes);
                        Assert.AreEqual(1, result.AllocatedNodes.Count, " Should have only run the task on 1 core but ran on "+ result.AllocatedNodes.Count + " cores (" + allocatedNodes + ")");
                    }
                }
    
                private static int SubmitJob(ISchedulerJob job, Scheduler scheduler, int numberOfTasks, int coresPerTask, int taskSleepInSeconds, string hpcUsername, string hpcPassword)
                {
                    for (int i = 0; i < numberOfTasks; i++)
                    {
                        ISchedulerTask task = job.CreateTask();
                        task.MinimumNumberOfCores = coresPerTask;
                        task.MaximumNumberOfCores = coresPerTask;
                        task.CommandLine = "powershell start-sleep " + taskSleepInSeconds + "";
                        job.AddTask(task);
                    }
                    scheduler.SubmitJob(job, hpcUsername, hpcPassword);
                    return job.Id;
                }
            }


    • Edited by TimJRoberts1 Friday, September 26, 2014 12:07 PM
    Friday, September 26, 2014 12:06 PM