No endpoint listening at net.tcp://<headnode>:5802/SchedulerStoreService


  • Hi,

    I'm getting the following exception:

    System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at net.tcp://<headnode>:5802/SchedulerStoreService that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details.

    The code throwing this exception (see snippet below) often executes without throwing an exception.

    var job = MyScheduler.OpenJob(jobId);
    job.Progress = percentageComplete;
    Has anyone seen this before? This method is being called a lot as our cluster often has many concurrently running Jobs that are having their progress properties updated. Is it possible the SchedulerStoreService cannot cope with several concurrent calls?

    Tuesday, July 10, 2018 10:09 AM

All replies

  • Hi Matt,

      SchedulerStoreService can cope with concurrent calls. From your description, this call sometime works sometime throw exception, right?

    Qiufang Shi

    Friday, July 13, 2018 4:02 AM
  • Yes - I'd say it works about 99% of the time, but it's receiving a lot of calls so even a 1% failure rate is quite a lot of failures!

    Friday, July 13, 2018 7:55 AM
  • Hi Matt,

      Could you share the version of HPC Pack you're using? And the load you have? The system currently don't have throttling design in place, thus under heavy load situation, calls may fail due to underlying SQL query/transaction failures.

    Qiufang Shi

    Monday, July 16, 2018 4:35 AM
  • Hi,

    My HPC Pack is HPC Pack 2016 v5.1.6086.0.

    An example of the load on the SchedulerStoreService is about 30 concurrently running HPC Jobs, each making regular progress update method calls as shown in my original post.

    Cheers, Matt.

    Monday, July 16, 2018 8:36 AM
  • Got it, will do local repro. and report back to this thread

    Qiufang Shi

    16 hours 44 minutes ago