none
RequestReplyCorrelator ArgumentException RRS feed

  • Question

  • [HPC Server 2008 R2 + HPC Pack RC1]

    Any idea where I should start with analyzing this one?

    System.ServiceModel.FaultException: Request failed 1 time(s) and is marked as failure by the broker. The exception is System.ArgumentException: Item has already been added. Key in dictionary: 'System.ServiceModel.Channels.RequestReplyCorrelator+Key: {urn:uuid:df1c9d3c-573e-496b-a25c-460009150e79, System.ServiceModel.Dispatcher.DuplexChannelBinder+IDuplexRequest}'  Key being added: 'System.ServiceModel.Channels.RequestReplyCorrelator+Key: {urn:uuid:df1c9d3c-573e-496b-a25c-460009150e79, System.ServiceModel.Dispatcher.DuplexChannelBinder+IDuplexRequest}'

    Server stack trace:
       at System.Collections.Hashtable.Insert(Object key, Object nvalue, Boolean add)
       at System.ServiceModel.Channels.RequestReplyCorrelator.System.ServiceModel.Channels.IRequestReplyCorrelator.Add[T](Message request, T state)
       at System.ServiceModel.Dispatcher.DuplexChannelBinder.RequestStarting(Message message, IDuplexRequest request)
       at System.ServiceModel.Dispatcher.DuplexChannelBinder.BeginRequest(Message message, TimeSpan timeout, AsyncCallback callback, Object state)
       at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.StartSend(Boolean completedSynchronously)
       at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.FinishEnsureOpen(IAsyncResult result, Boolean completedSynchronously)
       at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.StartEnsureOpen(Boolean completedSynchronously)
       at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.FinishEnsureInteractiveInit(IAsyncResult result, Boolean completedSynchronously)
       at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.StartEnsureInteractiveInit()
       at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.Begin()
       at System.ServiceModel.Channels.ServiceChannel.BeginCall(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, TimeSpan timeout, AsyncCallback callback, Object asyncState)
       at System.ServiceModel.Channels.ServiceChannelProxy.InvokeBeginService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
       at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

    Exception rethrown at [0]:
       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
       at Microsoft.Hpc.ServiceBroker.BackEnd.IService.BeginProcessMessage(Message request, AsyncCallback callback, Object asyncState)
       at Microsoft.Hpc.ServiceBroker.BackEnd.ServiceClient.BeginProcessMessage(Message request, AsyncCallback callback, Object asyncState)
       at Microsoft.Hpc.ServiceBroker.BackEnd.Dispatcher.ReceiveRequest(BrokerQueueItem item, Object state).

    Server stack trace:
       at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
       at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
       at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs)
       at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
       at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

    Exception rethrown at [0]:
       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
       at XXX.Grid.Hpc.HpcGridServiceReference.IHpcGridService.DoGridTask(DoGridTaskRequest request)
       at XXX.Grid.Hpc.HpcGridClient.SubmitJob(GridJob job, ProgressNotification progressNotifier) in C:\dev\GridRC1\trunk\GridComputing\XXX.Grid.Hpc\Provider\HpcGridClient.cs:line 103

     

    Every subsequent attempt to submit a request to this cluster gives (on the client):

    System.ServiceModel.FaultException: Request failed 1 time(s) and is marked as failure by the broker. The exception is System.ServiceModel.CommunicationException: The server did not provide a meaningful reply; this might be caused by a contract mismatch, a premature session shutdown or an internal server error.

    Server stack trace:
       at System.ServiceModel.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult result)
       at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
       at System.ServiceModel.Channels.ServiceChannelProxy.InvokeEndService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
       at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

    Exception rethrown at [0]:
       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
       at Microsoft.Hpc.ServiceBroker.BackEnd.IService.EndProcessMessage(IAsyncResult ar)
       at Microsoft.Hpc.ServiceBroker.BackEnd.ServiceClient.EndProcessMessage(IAsyncResult ar)
       at Microsoft.Hpc.ServiceBroker.BackEnd.Dispatcher.ResponseReceived(IAsyncResult ar).

    Server stack trace:
       at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc& rpc)
       at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
       at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs)
       at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
       at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

    Exception rethrown at [0]:
       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
       at XXX.Grid.Hpc.HpcGridServiceReference.IHpcGridService.DoGridTask(DoGridTaskRequest request)
       at XXX.Grid.Hpc.HpcGridClient.SubmitJob(GridJob job, ProgressNotification progressNotifier) in C:\dev\GridRC1\trunk\GridComputing\XXX.Grid.Hpc\Provider\HpcGridClient.cs:line 103

     

    (and in the svclog):

    <E2ETraceEvent xmlns="http://schemas.microsoft.com/2004/06/E2ETraceEvent"><System xmlns="http://schemas.microsoft.com/2004/06/windows/eventlog/system"><EventID>131075</EventID><Type>3</Type><SubType Name="Error">0</SubType><Level>2</Level><TimeCreated SystemTime="2010-07-19T20:27:58.7272907Z" /><Source Name="System.ServiceModel" /><Correlation ActivityID="{2286dd48-43d1-4993-9593-d4b2ea31a0d6}" /><Execution ProcessName="HpcServiceHost32" ProcessID="12" ThreadID="6" /><Channel /><Computer>WBRADNEYHPCRC1</Computer></System><ApplicationData><TraceData><DataItem><TraceRecord xmlns="http://schemas.microsoft.com/2004/10/E2ETraceEvent/TraceRecord" Severity="Error"><TraceIdentifier>http://msdn.microsoft.com/en-US/library/System.ServiceModel.Diagnostics.ThrowingException.aspx</TraceIdentifier><Description>Throwing an exception.</Description><AppDomain>ServiceDomain</AppDomain><Exception><ExceptionType>System.ServiceModel.CommunicationException, System.ServiceModel, Version=3.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType><Message>The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:00:04.1759675'.</Message><StackTrace>   at System.ServiceModel.Channels.SocketConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)
       at System.ServiceModel.Channels.BufferedConnection.WriteNow(Byte[] buffer, Int32 offset, Int32 size, TimeSpan timeout, BufferManager bufferManager)
       at System.ServiceModel.Channels.BufferedConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)
       at System.ServiceModel.Channels.DelegatingConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)
       at System.ServiceModel.Channels.TracingConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)
       at System.ServiceModel.Channels.InitialServerConnectionReader.SendFault(IConnection connection, String faultString, Byte[] drainBuffer, TimeSpan sendTimeout, Int32 maxRead)
       at System.ServiceModel.Channels.ServerSessionPreambleConnectionReader.SendFault(String faultString)
       at System.ServiceModel.Channels.ServerSessionPreambleConnectionReader.ContinueReading()
       at System.ServiceModel.Channels.ServerSessionPreambleConnectionReader.StartReading(OnViaDelegate viaDelegate, TimeSpan receiveTimeout)
       at System.ServiceModel.Channels.ConnectionDemuxer.OnDuplexConnection(IConnection connection, ItemDequeuedCallback connectionDequeuedCallback, Int64 streamPosition, Int32 offset, Int32 size, TimeSpan timeout)
       at System.ServiceModel.Channels.ConnectionDemuxer.OnConnectionModeKnownCore(ConnectionModeReader modeReader, Boolean isCached)
       at System.ServiceModel.Channels.ConnectionDemuxer.OnConnectionModeKnown(ConnectionModeReader modeReader)
       at System.ServiceModel.Channels.ConnectionModeReader.Complete()
       at System.ServiceModel.Channels.ConnectionModeReader.ReadCallback(Object state)
       at System.ServiceModel.Channels.TracingConnection.TracingConnectionState.ExecuteCallback()
       at System.ServiceModel.Channels.TracingConnection.WaitCallback(Object state)
       at System.ServiceModel.Channels.SocketConnection.FinishRead()
       at System.ServiceModel.Channels.SocketConnection.AsyncReadCallback(Boolean haveResult, Int32 error, Int32 bytesRead)
       at System.ServiceModel.Channels.OverlappedContext.CompleteCallback(UInt32 error, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
       at System.ServiceModel.Diagnostics.Utility.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
       at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
    </StackTrace><ExceptionString>System.ServiceModel.CommunicationException: The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:00:04.1759675'. ---&amp;gt; System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
       at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
       at System.ServiceModel.Channels.SocketConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)
       --- End of inner exception stack trace ---</ExceptionString><InnerException><ExceptionType>System.Net.Sockets.SocketException, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089</ExceptionType><Message>An existing connection was forcibly closed by the remote host</Message><StackTrace>   at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
       at System.ServiceModel.Channels.SocketConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)</StackTrace><ExceptionString>System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host
       at System.Net.Sockets.Socket.Send(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags)
       at System.ServiceModel.Channels.SocketConnection.Write(Byte[] buffer, Int32 offset, Int32 size, Boolean immediate, TimeSpan timeout)</ExceptionString><NativeErrorCode>2746</NativeErrorCode></InnerException></Exception></TraceRecord></DataItem></TraceData></ApplicationData></E2ETraceEvent>

    Monday, July 19, 2010 7:36 PM

Answers

All replies

  • Hi,

    This looks mostly like a bug we fixed after RC1. Can you hold on for RTM and tried this again?

    -yiding

    Wednesday, July 28, 2010 9:14 AM
  • That depends -- when is RTM coming out?

    I'm concerned that I haven't been able to get this particular scenario working in RC1, when it was working just fine in Beta 1.

    Any chance of an RC2?

    I'm also happy to work directly with someone on the dev team to investigate.

    Wednesday, July 28, 2010 1:31 PM
  • Windows HPC Server 2008 R2 Express (full) and Enterprise+HPC Edition OS (evaluation) are available on the download center already.

    For TAP customers who have already agreed to purchase Enterprise, or have a software assurance subscription, a licensed version of Enterprise is available on the Connect site.

    Customers with MSDN subscriptions can download Enterprise and the HPC Edition operating system through their MSDN page.

    http://blogs.technet.com/b/windowshpc/archive/2010/07/26/hpc-pack-2008-r2-and-the-windows-hpc-server-2008-r2-suite-have-been-released.aspx

     

    Tuesday, August 3, 2010 5:47 AM
  • There is no RC2 for HPC. It'll be great if you can try out the RTM version and see if the bug is gone. If you still can reproduce the bug, the product team will be happy to work with you on this one.
    Tuesday, August 3, 2010 5:50 AM