none
SOA Job - Failed state - But works ok RRS feed

  • Question

  • Lately, am seeing my SOA jobs end with failed requests = number of requests.

    Oddly, what ever the job is supposed to do, its working just fine.

    environment: HPC Pack 2019

    client and server version: 6.0.7205.0 

    head node: windows server 2016

    SOA job trace has following info (for a COMPUTE NODE on 2012 Win server). Otherwise, everything looks ok. Certificate etc. Whats missing?

    Dispatch history #0 (Id 57c3a2a5-07a1-4056-b53d-5064903bae87):
    Dispatch time: 11/12/2020 1:34:27 PM
    Response Time: 11/12/2020 1:34:29 PM
    Response Type: CommunicationError
    Target Machine: COMPUTE-NODE
    Task Id: 6963
    Exception details: System.ServiceModel.CommunicationException: The server did not provide a meaningful reply; this might be caused by a contract mismatch, a premature session shutdown or an internal server error.

    Server stack trace: 
       at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult result)
       at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
       at System.ServiceModel.Channels.ServiceChannelProxy.InvokeEndService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
       at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

    Exception rethrown at [0]: 
       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
       at Microsoft.Hpc.ServiceBroker.BackEnd.IService.EndProcessMessage(IAsyncResult ar)
       at Microsoft.Hpc.ServiceBroker.BackEnd.ResponseReceiver.ReceiveResponse(DispatchData data)
    User traces in this dispatch
    Trace Level  Trace Text  Trace Time  Process Thread
    5 [HpcServiceHost]: Request is received. 11/12/2020 1:35:28 PM 2552 292
    5 [HpcServiceHost]: Response is sent back. IsFault = False 11/12/2020 1:35:30 PM 2552 292
    Dispatch history #1 (Id 29dbf8f0-ad6b-484d-a3bb-3c35eb780e9f):
    Dispatch time: 11/12/2020 1:34:26 PM
    Response Time: 11/12/2020 1:34:27 PM
    Response Type: CommunicationError
    Target Machine: COMPUTE-NODE
    Task Id: 6975
    Exception details: System.ServiceModel.CommunicationException: The server did not provide a meaningful reply; this might be caused by a contract mismatch, a premature session shutdown or an internal server error.

    Server stack trace: 
       at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult result)
       at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
       at System.ServiceModel.Channels.ServiceChannelProxy.InvokeEndService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
       at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

    Exception rethrown at [0]: 
       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
       at Microsoft.Hpc.ServiceBroker.BackEnd.IService.EndProcessMessage(IAsyncResult ar)
       at Microsoft.Hpc.ServiceBroker.BackEnd.ResponseReceiver.ReceiveResponse(DispatchData data)
    User traces in this dispatch
    Trace Level  Trace Text  Trace Time  Process Thread
    5 [HpcServiceHost]: Request is received. 11/12/2020 1:35:27 PM 276 2212
    5 [HpcServiceHost]: Response is sent back. IsFault = False 11/12/2020 1:35:28 PM 276 2212

    Thursday, November 12, 2020 8:31 PM

All replies

  • Hi SRIRAM R,

    It looks a service side error. Could you check your service code if there is any log? Do all session jobs failed the same way? Could you run the built-in Echo service by EchoClient.exe and see if it works fine?

    Meanwhile, are all nodes domain joined? Do you have any specific configuration in the service registration file for broker backend binding?

    Regards,

    Yutong Sun

    Tuesday, November 24, 2020 11:55 AM
    Moderator