none
HPC SOA Job behavior RRS feed

  • Question

  • Hi Everyone 

    I have installed Microsoft HPC 2016. I have this situation - after submission SOA job it started ~17mb and growing memory. I submitted different job with different configuration (messageThrottleStartThreshold="256", messageThrottleStopThreshold="128" and  maxMessageSize="32768"). It did not help. All time it is growing. I submitted CcpEchoSvc job with default configuration (it is in HPC) and it is growing too. During two day it took 624MB (it was idl, my job did not do anything). I have noticed the SOA job the first period of life growing memory fast , but after some times it become slowly.

    What you will suggest for SOA jobs? What configuration do I need?  Will you suggest to have long running SOA jobs or you have alternative solution for that ? 

    Thanks

    Artem

    Monday, October 23, 2017 9:39 AM

All replies

  • Hi Artem,

    For the growing memory, is it refer to the HpcBrokerWorker.exe on the broker node which acts as the real broker to handle SOA requests/responses? The message throttling configurations should act on this broker process to limit the memory consumption. Also note there are three idle broker workers on a broker node, and the broker launcher (HpcBroker.exe) would pick up one to handle the SOA session created. After the session is closed, the according HpcBrokerWorker.exe would exit. The session launcher (HpcSession.exe) on the head node and the broker launcher (HpcBroker.exe) on the broker node are long running service/process compared to the broker worker. If you have a high spec machine with large memory size, these services can consume a lot of free memory e.g. 1GB when the GC is slow on the managed heap. No worry if the memory consumptions for these services are not growing constantly.

    Long running SOA jobs is OK and should be handled nicely on the SOA runtime. Just note the timeout settings e.g.  serviceOperationTimeout/clientIdleTimeout/sessionIdleTimeout for the long running SOA services.

    Regards,

    Yutong Sun

    Tuesday, October 24, 2017 1:39 AM
  • Hi Yutong

    When I submitted job it is started from ~17MB and growing to around ~650MB (I follow HpcBrokerWorker.exe) . I notice it is growing till around ~650(+/-10MB) and after it become around ~550. Please confirm that this behavior is because of Garbage Collector.

    Thanks

    Artem

    Tuesday, October 24, 2017 7:47 AM
  • Hi Yutong,

    Artem and I are working in the same team and we have successfully migrated to HPC for our batch processing capabilities. Now, we have to support interactive jobs with HPC, and these interactive jobs sessions can be opened for a very long period of time (many days).

    We have concerns over this increasing memory as few months backe we did one POC with windows hpc 2012, and observed huge memory usage by broker node. Due to which our head node became unusable (we were running head-node and broker node on the same machine). Are there any specific memory settings we need to be aware of?

    Based on your answer, memory shouldn't be an issue & SOA jobs can support long running jobs.

    @Artem: Can you please try to recreate the high memory usage scenario (what we observed previously), and check if we get the same. After that we can share more details.

    Thanks,

    Puneet


    Puneet Sharma

    Tuesday, October 24, 2017 2:05 PM
  • Hi Artem,

    If you are tracing HpcBrokerWorker.exe, you may also check the number and data size of the SOA requests in the corresponding SOA session. For an interactive session, all requests and responses are hold in memory of HpcBrokerWorker.exe, which could be the cause of the high memory usage.

    Regards,

    Yutong Sun

    Wednesday, October 25, 2017 8:36 AM
  • Hi Puneet,

    Besides the throttling configurations per servcie, you may also limit the number of concurrent SOA sessions/HpcBrokerWorker.exe processes per broker node via the MaxConcurrentSession setting in HpcBroker.exe.config (HpcBroker service needs restart after the change), so that the broker node can serve a proper number of concurrent SOA sessions according to its hardware spec (besides Mem, CPU/Disk/Network may also be considered).

    Regards,

    Yutong Sun

    Wednesday, October 25, 2017 8:45 AM
  • Hi Yutong

    Is it possible to configure keeping requests and responses in memory of HpcBrokerWorker.exe. For example keep requests and responses only for last 5 minutes or last 100 MB.

    Thanks

    Artem

    Thursday, October 26, 2017 11:16 AM
  • Hi Artem,

    You may use throttling configurations (messageThrottleStartThreshold/messageThrottleStopThreshold) to limit the number of requests in the memory of the broker worker. As for responses of an interactive session, once they are successfully retrieved by the client, they will be purged from the memory. You may also call BrokerClient.Close(purge:true) at the client to remove the requests/responses from the broker worker.

    For the original memory consumption issue of an idle HpcBrokerWorker.exe, I had a similar repro locally and will investigate it further. Thanks for reporting this.

    Regards,

    Yutong Sun

    Friday, October 27, 2017 8:34 AM