locked
Submitting multiple jobs in parallel over HTTPS with latency scales poorly in HPC Pack 2012 R2 RRS feed

  • Question

  • I have found that submitting multiple jobs in parallel via HTTPS does not scale well when there is network latency between client and headnode.

    I wrote a simple test to create/addTask/submit jobs in parallel from a client using .Net SDK. As I increase the number of parallel jobs submitted at once the increase is linear.

    I ran the same test under the following circumstances. "far from" = Europe to North America.

    • client is far from headnode over HTTPS = BAD
    • client is far from headnode using windowsauthentication = GOOD
    • client is near headnode using HTTPS = GOOD
    • client is near headnode using windowsauthentication = GOOD

    Here's the results I see: Any idea why HTTPS is so bad when there is network latency? 

    Any ways to improve this? Apart from switching to WindowsAuthentication or moving the client nearer to the headnode.

    Using version 4.5.5079.0 client and server.

    Thursday, April 27, 2017 10:51 AM

Answers

  • Hi TimJRoberts1,

    Thanks for you advice. We changed the setting of DefaultConnectionLimit in client code before connect to HPC Pack 2016 cluster like following according to your post:

    ServicePointManager.DefaultConnectionLimit = 10;
    var scheduler = new Scheduler();
    scheduler.Connect("headnode");

    And below is the test result. After changing this configuration, https scales as well as net.tcp.

    For now we would suggest you (and your clients) add this configuration before connecting to scheduler as a workaround. We will fix that in our future release.

    Thanks again,
    Zihao


    Tuesday, May 16, 2017 4:05 AM

All replies

  • This sounds like there is a bug in this situation "client is far from headnode over HTTPS = BAD". Could you share the client code you do the testing? we can take a check on that.

    Qiufang Shi

    Friday, April 28, 2017 10:33 AM
  •     internal class Program
        {
            private static void Main(string[] args)
            {
                if (args == null || args.Length != 4)
                {
                    Console.WriteLine("incorrect usage");
                    Console.WriteLine("expected " + Assembly.GetExecutingAssembly().GetName().Name + " [headnode] [numberOfJobs] [numberOfTasksPerJob] [secondsToSleep]");
                    Console.WriteLine("e.g.");
                    Console.WriteLine(Assembly.GetExecutingAssembly().GetName().Name + " myheadnode 5 4 3");
                    return;
                }
                var scheduler = new Scheduler();
                var headnode = args[0];
                scheduler.Connect(headnode);
                int numberOfJobs = int.Parse(args[1]);
                int numberOfTasksPerJob = int.Parse(args[2]);
                int secondsToSleep = int.Parse(args[3]);
                Console.WriteLine("submit " + numberOfJobs + " jobs with " + numberOfTasksPerJob + " tasksPerJob");
                var objlock = new object();
    
                var finishedResetEvents = new Dictionary<int, ManualResetEvent>();
                Parallel.For(0, numberOfJobs, i =>
                {
                    var fulltimer = Stopwatch.StartNew();
                    var timer = Stopwatch.StartNew();
                    var job = scheduler.CreateJob();
                    Console.WriteLine("CreateJob " + job.Id + " took " + timer.ElapsedMilliseconds);
                    timer.Restart();
    
                    scheduler.AddJob(job);
                    Console.WriteLine("AddJob " + job.Id + " took " + timer.ElapsedMilliseconds);
                    var jobId = job.Id;
                    var task = job.CreateTask();
                    task.Type = TaskType.ParametricSweep;
                    task.StartValue = 0;
                    task.EndValue = numberOfTasksPerJob;
                    string commandLine = "powershell -command \"Start-Sleep " + secondsToSleep + "\"";
                    task.CommandLine = commandLine;
                    job.AddTask(task);
                    var jobFinished = new ManualResetEvent(false);
                    lock (objlock)
                    {
                        finishedResetEvents.Add(jobId, jobFinished);
                    }
                    timer.Restart();
                    job.OnJobState += (sender, arg) =>
                    {
                        if (arg.NewState == JobState.Finished || arg.NewState == JobState.Failed)
                        {
                            jobFinished.Set();
                        }
                    };
                    Console.WriteLine("OnJobState+= " + job.Id + " took " + timer.ElapsedMilliseconds);
                    timer.Restart();
                    job.OnTaskState += (sender, arg) => { Console.WriteLine("task: " + arg.TaskId.JobTaskId + " newstate: " + arg.NewState);};
                    Console.WriteLine("OnTaskState+= " + job.Id + " took " + timer.ElapsedMilliseconds);
                    timer.Restart();
                    scheduler.SubmitJob(job, null, null);
                    Console.WriteLine("SubmitJob " + job.Id + " took " + timer.ElapsedMilliseconds);
                    Console.WriteLine("Total submit time for job: " + job.Id + " took " + fulltimer.ElapsedMilliseconds);
                });
    
                var finished = WaitHandle.WaitAll(finishedResetEvents.Values.Cast<WaitHandle>().ToArray(), TimeSpan.FromSeconds(300));
                Console.WriteLine("finished: " + finished);
            }
        }

    I just ran this on the commandline as

    .\HpcTestClient.exe MYHEADNODEHOSTNAME 1 6 2

    .\HpcTestClient.exe MYHEADNODEHOSTNAME 2 6 2

    .\HpcTestClient.exe MYHEADNODEHOSTNAME 4 6 2

    .\HpcTestClient.exe MYHEADNODEHOSTNAME 8 6 2

    .\HpcTestClient.exe MYHEADNODEHOSTNAME 16 6 2

    .\HpcTestClient.exe MYHEADNODEHOSTNAME 32 6 2

    then again as .\HpcTestClient.exe https://MYHEADNODEHOSTNAME 1 6 2

    .. etc etc

    and ran the same console app in various locations, to get different latencies between client and headnode.PS my "client" was a 32 core server, so Parallel.For ran all at once all the way up to 32 jobs.

    The chart above was made from max duration for each "Total submit time for job: " result per run.


    • Edited by TimJRoberts1 Friday, April 28, 2017 12:20 PM removed unit test assertion
    Friday, April 28, 2017 12:18 PM
  • Were you able to reproduce? It behaves as though something is locking in the HTTPS client or server, making it single threaded.
    Friday, May 5, 2017 8:57 AM
  • Hi TimJRoberts1,

    We are still trying to reproduce your issue. Thanks for your patience.

    Thanks,
    Zihao


    Wednesday, May 10, 2017 1:27 AM
  • Hi TimJRoberts1,

    We did some test using the code sample you provided, and get the result attached below, in which "client is near headnode using HTTPS" is also bad.
    When you choose to connect to headnode with windows authentication, we leverage .net remoting + keberos to accomplish that. When you try to connect though https, we use WCF + http binding, and http binding is expected to be slower. So the following result meets our expectation.

    Could you tell us your user scenario? We can provide more specific advice if we know more about the reason you need fast parallel job submitting through https.

    Thanks,
    Zihao


    Wednesday, May 10, 2017 5:54 AM
  • Thanks for investigating!

    My scenario is I provide an abstraction to HPC for many grid users. The abstraction runs on-premise to allow integration with enterprise Active Directory.

    The HPC cluster is in AWS (cannot switch to Azure).

    I cannot create a trusted relationship between AWS and on-premise domains. Therefore I can only use HTTPS.

    Some of the internal clients to my abstraction submit many jobs at once. This is where I noticed the slowdown.

    I understand there is overhead for HTTPS/WCF, but am surprised that it scales as though it is only using 1 thread.

    i.e. why does submitting two jobs in parallel via HTTPS take twice as long as one (when you have many cores and network bandwidth available)? Kerberos works as expected.

    Thursday, May 11, 2017 9:30 AM
  • Hi TimJRoberts1,

    We did more tests, and found out the problem is caused by BasicHttpBinding and perhaps our code interacting with it. Test result is attached at the end of post in case you are interested. We'll continue to investigate and inform you if we have any progress.

    Back to your scenario, is there any resource (i.e. core, memory) wasted because of the slow job submitting rate in your cluster? If yes, we'll advice you to
    1. Check if there are clients who submit multiple jobs at once. Advice them to submit one job and multiple tasks instead. As there is scheduler api which can help you create and add multiple tasks at once, and also task is more light-weight than job, this'll shorten job submitting time.
    2. Use AutoGrowShrink. If the jobs are relatively small, and due to the submitting rate it is not possible to utilize all the resources all the time, then please consider AutoGrowShrink. We officially support AutoGrowShrink only in Azure. You can check if there is an alternative way doing this in AWS.

    Thanks,
    Zihao

    About the test result: We tested submitting 1~64 jobs with 6 tasks in each and we tested in HPC Pack 2012R2 and HPC Pack 2016. In both case, submitting jobs through https only scales up to two threads. As in HPC Pack 2016 we replace .net remoting with WCF, we believe this is caused by BasicHttpBinding.

    Monday, May 15, 2017 5:42 AM
  • Thanks again for your effort to investigate. Possibly the default value for ServicePointManager.DefaultConnectionLimit is causing the limit of 2?

    https://msdn.microsoft.com/en-us/library/system.net.servicepointmanager.defaultconnectionlimit(v=vs.110).aspx

    Monday, May 15, 2017 2:08 PM
  • Hi TimJRoberts1,

    Thanks for you advice. We changed the setting of DefaultConnectionLimit in client code before connect to HPC Pack 2016 cluster like following according to your post:

    ServicePointManager.DefaultConnectionLimit = 10;
    var scheduler = new Scheduler();
    scheduler.Connect("headnode");

    And below is the test result. After changing this configuration, https scales as well as net.tcp.

    For now we would suggest you (and your clients) add this configuration before connecting to scheduler as a workaround. We will fix that in our future release.

    Thanks again,
    Zihao


    Tuesday, May 16, 2017 4:05 AM
  • Great thanks I also see the same behaviour when increasing DefaultConnectionLimit
    Tuesday, May 16, 2017 10:33 AM