Lately, am seeing my SOA jobs end with failed requests = number of requests.
Oddly, what ever the job is supposed to do, its working just fine.
environment: HPC Pack 2019
client and server version: 6.0.7205.0
head node: windows server 2016
SOA job trace has following info (for a COMPUTE NODE on 2012 Win server). Otherwise, everything looks ok. Certificate etc. Whats missing?
Dispatch history #0 (Id 57c3a2a5-07a1-4056-b53d-5064903bae87):
Dispatch time: 11/12/2020 1:34:27 PM
Response Time: 11/12/2020 1:34:29 PM
Response Type: CommunicationError
Target Machine: COMPUTE-NODE
Task Id: 6963
Exception details: System.ServiceModel.CommunicationException: The server did not provide a meaningful reply; this might be caused by a contract mismatch, a premature session shutdown or an internal server error.
Server stack trace:
at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeEndService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.Hpc.ServiceBroker.BackEnd.IService.EndProcessMessage(IAsyncResult ar)
at Microsoft.Hpc.ServiceBroker.BackEnd.ResponseReceiver.ReceiveResponse(DispatchData data)
User traces in this dispatch
Trace Level Trace Text Trace Time Process Thread
5
[HpcServiceHost]: Request is received.
11/12/2020 1:35:28 PM
2552
292
5
[HpcServiceHost]: Response is sent back. IsFault = False
11/12/2020 1:35:30 PM
2552
292
Dispatch history #1 (Id 29dbf8f0-ad6b-484d-a3bb-3c35eb780e9f):
Dispatch time: 11/12/2020 1:34:26 PM
Response Time: 11/12/2020 1:34:27 PM
Response Type: CommunicationError
Target Machine: COMPUTE-NODE
Task Id: 6975
Exception details: System.ServiceModel.CommunicationException: The server did not provide a meaningful reply; this might be caused by a contract mismatch, a premature session shutdown or an internal server error.
Server stack trace:
at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeEndService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.Hpc.ServiceBroker.BackEnd.IService.EndProcessMessage(IAsyncResult ar)
at Microsoft.Hpc.ServiceBroker.BackEnd.ResponseReceiver.ReceiveResponse(DispatchData data)
User traces in this dispatch
Trace Level Trace Text Trace Time Process Thread
5
[HpcServiceHost]: Request is received.
11/12/2020 1:35:27 PM
276
2212
5
[HpcServiceHost]: Response is sent back. IsFault = False
11/12/2020 1:35:28 PM
276
2212
