locked
GetChangeBatch TimeOut RRS feed

  • Question

  • I am using the 2.1 Sync Framework to sync a Sql Compact database with a SQL Server database. I have a proxy that I run through since it's an N-Tier enviornment.

    I'm batching with a size of 4000 (not sure if that's huge by standards or not). I am syncing down 5 different scopes one at a time for the initial sync. It works just fine for all of them but my history and I think it has something to do with size. 3 of the 5 are rather large scaling from 15mb to 80mb in size. One of them is only one row at the moment and it doesn't have any problems. The History is about 1.7 mb in size for this given customer and it's not returning my call to GetChanges.

    I ran it in local mode and turned on Sync Tracing which doesn't return anything other then the EndPoint address could not do anything with the message that was returned. It gets the changesWrapper (GetChagnesParameters) just fine and it checks that DBSyncContext to see it it should  batch which it doesn't. After the return call for the changesWrapper it just sits there and times out. I noticed that the History and the other small scope did not batch which was to be expected since they are much smaller then my batch size. I adjusted the batch size down to 1000 and the history started working again. I set it back up to say 2000 and it stops working.  I checked my timeout which is set to 5 minutes (really high I know but it's still in development and some customers have dial up (yuck)).

    I have ran this in a Peer to Peer enviornment and it works just fine.

    So the question is this. Is there a thresh hold as to the batch file size or how much can be returned while not batching? More information is needed but this is a start.

    Here is the function from my Proxy

      Public Overrides Function GetChangeBatch(ByVal batchSize As UInteger, ByVal destinationKnowledge As Microsoft.Synchronization.SyncKnowledge, ByRef changeDataRetriever As Object) As Microsoft.Synchronization.ChangeBatch
        Try
          If scopeName.ToLower = "history" Then
            Dim x As String = ""
          End If
          Dim changesWrapper As GetChangesParameters = proxy.GetChanges(batchSize, destinationKnowledge)
          'Retrieve the ChangeDataRetriever and the ChangeBatch
          changeDataRetriever = changesWrapper.DataRetriever
    
          Dim context As DbSyncContext = TryCast(changeDataRetriever, DbSyncContext)
          'Check to see if the data is batched.
          If context IsNot Nothing AndAlso context.IsDataBatched Then
            If Me.localBatchingDirectory Is Nothing Then
              'Retrieve the remote peer id from the MadeWithKnowledge.ReplicaId. MadeWithKnowledge is the local knowledge of the peer 
              'that is enumerating the changes.
              Dim remotePeerId As String = context.MadeWithKnowledge.ReplicaId.ToString()
    
              'Generate a unique Id for the directory.
              'We use the peer id of the store enumerating the changes so that the local temp directory is same for a given source
              'across sync sessions. This enables us to restart a failed sync by not downloading already received files.
              Dim sessionDir As String = Path.Combine(Me.m_batchingDirectory, "WebSync_" & remotePeerId)
              Me.localBatchingDirectory = New DirectoryInfo(sessionDir)
              'Create the directory if it doesnt exist.
              If Not Me.localBatchingDirectory.Exists Then
                Me.localBatchingDirectory.Create()
              End If
            End If
    
            Dim localFileName As String = Path.Combine(Me.localBatchingDirectory.FullName, context.BatchFileName)
            Dim localFileInfo As New FileInfo(localFileName)
    
            'Download the file only if doesnt exist        
            If Not localFileInfo.Exists Then
              Dim remoteFileContents As Byte() = Me.proxy.DownloadBatchFile(context.BatchFileName)
              Using localFileStream As New FileStream(localFileName, FileMode.Create, FileAccess.Write)
                localFileStream.Write(remoteFileContents, 0, remoteFileContents.Length)
              End Using
            End If
            'Set DbSyncContext.Batchfile name to the new local file name
            context.BatchFileName = localFileName
          End If
    
          Return changesWrapper.ChangeBatch
        Catch ex As Exception
          Dim a As String = ex.Message
          Dim b As String = ex.StackTrace
          Dim c As String = ""
          Dim d As String = ""
    
          If Not ex.InnerException Is Nothing Then
            c = ex.InnerException.Message
            d = ex.InnerException.StackTrace
          End If
    
        End Try
      
      End Function
    

    Here is from my Service

      Public Function GetChanges(ByVal batchSize As UInteger, ByVal destinationKnowledge As Microsoft.Synchronization.SyncKnowledge) As GetChangesParameters Implements IDynamicSyncContract.GetChanges
        Try
          Dim changesWrapper As New GetChangesParameters()
          changesWrapper.ChangeBatch = Me.peerProvider.GetChangeBatch(batchSize, destinationKnowledge, changesWrapper.DataRetriever)
    
          Dim context As DbSyncContext = TryCast(changesWrapper.DataRetriever, DbSyncContext)
          'Check to see if data is batched
          If context IsNot Nothing AndAlso context.IsDataBatched Then
            'Dont send the file location info. Just send the file name
            Dim fileName As String = New FileInfo(context.BatchFileName).Name
            Me.batchIdToFileMapper(fileName) = context.BatchFileName
            context.BatchFileName = fileName
          End If
          Return changesWrapper
        Catch ex As Exception
          Dim a As String = ex.Message
          Dim b As String = ex.StackTrace
          Dim c As String = ""
          Dim d As String = ""
    
          If Not ex.InnerException Is Nothing Then
            c = ex.InnerException.Message
            d = ex.InnerException.StackTrace
          End If
        End Try
    
      End Function
    

     

    Monday, March 7, 2011 7:20 PM

All replies

  • you might want to try enabling WCF Tracing to check if WCF shows something that needs to be tweaked around config or find out any exception...

    Tuesday, March 8, 2011 4:44 AM
  • The WCF Trace Log is throwing this error:

    The specified network name is no longer available. This often indicates that the HTTP client has prematurely closed the underlying TCP connection.

    My Stack Trace:

    System.ServiceModel.Channels.HttpOutput.ListenerResponseHttpOutput.ListenerResponseOutputStream.Write(Byte[] buffer, Int32 offset, Int32 count)
    System.ServiceModel.Channels.HttpOutput.Send(TimeSpan timeout)
    System.ServiceModel.Channels.HttpRequestContext.OnReply(Message message, TimeSpan timeout)
    System.ServiceModel.Channels.RequestContextBase.Reply(Message message, TimeSpan timeout)
    System.ServiceModel.Security.SecuritySessionServerSettings.SecuritySessionRequestContext.OnReply(Message message, TimeSpan timeout)
    System.ServiceModel.Channels.RequestContextBase.Reply(Message message, TimeSpan timeout)
    System.ServiceModel.Channels.ReliableChannelBinder`1.RequestRequestContext.OnReply(Message message, TimeSpan timeout)
    System.ServiceModel.Channels.RequestContextBase.Reply(Message message, TimeSpan timeout)
    System.ServiceModel.Channels.ReliableReplySessionChannel.ReliableRequestContext.SendReply(RequestContext context, MaskingMode maskingMode, TimeoutHelper& timeoutHelper)
    System.ServiceModel.Channels.ReliableReplySessionChannel.ReliableRequestContext.ReplyInternal(Message reply, TimeSpan timeout)
    System.ServiceModel.Channels.ReliableReplySessionChannel.ReliableRequestContext.OnReply(Message reply, TimeSpan timeout)
    System.ServiceModel.Channels.RequestContextBase.Reply(Message message, TimeSpan timeout)
    System.ServiceModel.Channels.TransactionRequestContext.OnReply(Message message, TimeSpan timeout)
    System.ServiceModel.Channels.RequestContextBase.Reply(Message message, TimeSpan timeout)
    System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.Reply(MessageRpc& rpc)
    System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage9(MessageRpc& rpc)
    System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage8(MessageRpc& rpc)
    System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
    System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage41(MessageRpc& rpc)
    System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(MessageRpc& rpc)
    System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)
    System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage3(MessageRpc& rpc)
    System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage2(MessageRpc& rpc)
    System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(MessageRpc& rpc)
    System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage1(MessageRpc& rpc)
    System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
    System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(RequestContext request, Boolean cleanThread, OperationContext currentOperationContext)
    System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext request, OperationContext currentOperationContext)
    System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult result)
    System.ServiceModel.Dispatcher.ChannelHandler.OnAsyncReceiveComplete(IAsyncResult result)
    System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
    System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
    System.ServiceModel.Channels.ReceiveTimeoutAsyncResult.Callback(IAsyncResult result)
    System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
    System.ServiceModel.Diagnostics.TraceUtility.<>c__DisplayClass4.<CallbackGenerator>b__2(AsyncCallback callback, IAsyncResult result)
    System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
    System.Runtime.InputQueue`1.AsyncQueueReader.Set(Item item)
    System.Runtime.InputQueue`1.Dispatch()
    System.ServiceModel.Channels.ReliableReplySessionChannel.ProcessSequencedMessage(RequestContext context, String action, WsrmSequencedMessageInfo info)
    System.ServiceModel.Channels.ReliableReplySessionChannel.ProcessRequest(RequestContext context, WsrmMessageInfo info)
    System.ServiceModel.Channels.ReliableReplySessionChannel.HandleReceiveComplete(IAsyncResult result)
    System.ServiceModel.Channels.ReliableReplySessionChannel.OnReceiveCompletedStatic(IAsyncResult result)
    System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
    System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
    System.Runtime.AsyncResult.Complete(Boolean completedSynchronously, Exception exception)
    System.ServiceModel.Channels.ReliableChannelBinder`1.InputAsyncResult`1.OnInputComplete(IAsyncResult result)
    System.ServiceModel.Channels.ReliableChannelBinder`1.InputAsyncResult`1.OnInputCompleteStatic(IAsyncResult result)
    System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
    System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
    System.Runtime.AsyncResult.Complete(Boolean completedSynchronously, Exception exception)
    System.ServiceModel.Security.SecuritySessionServerSettings.ServerSecuritySessionChannel.ReceiveRequestAsyncResult.Complete(Boolean synchronous, Exception exception)
    System.ServiceModel.Security.SecuritySessionServerSettings.ServerSecuritySessionChannel.ReceiveRequestAsyncResult.OnReceive(IAsyncResult result)
    System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
    System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
    System.Runtime.AsyncResult.Complete(Boolean completedSynchronously, Exception exception)
    System.ServiceModel.Channels.ReliableChannelBinder`1.InputAsyncResult`1.OnInputComplete(IAsyncResult result)
    System.ServiceModel.Channels.ReliableChannelBinder`1.InputAsyncResult`1.OnInputCompleteStatic(IAsyncResult result)
    System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
    System.ServiceModel.Diagnostics.TraceUtility.<>c__DisplayClass4.<CallbackGenerator>b__2(AsyncCallback callback, IAsyncResult result)
    System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
    System.Runtime.InputQueue`1.AsyncQueueReader.Set(Item item)
    System.Runtime.InputQueue`1.Dispatch()
    System.Runtime.InputQueue`1.OnDispatchCallback(Object state)
    System.Runtime.ActionItem.DefaultActionItem.Invoke()
    System.Runtime.ActionItem.CallbackHelper.InvokeWithoutContext(Object state)
    System.Runtime.IOThreadScheduler.ScheduledOverlapped.IOCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
    System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
    System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)
    

    From what I can read on this one is that it's usally the client so I went and set all the timeouts to make sure this doesn't happen. What's really interesting is that it throws this about 30 seconds from the start of the log which makes me think hey it's a default somewhere set to 30 sec. But as far as I can tell nothing is.

    It's still strange two of the scopes require multiple batch files (1 requires about 10 or so) and they have no problems. This one isn't big enough to bother batching but it can't return the data.

    Here is the bindings portion for the server

     <bindings>
      <wsHttpBinding>
      <!-- We are using Server cert only.-->
      <binding name="largeMessageHttpBinding" transactionFlow="True" openTimeout ="00:05:00" closeTimeout ="00:05:00" receiveTimeout="00:05:00" sendTimeout="00:05:00" maxReceivedMessageSize="999999999">
       <readerQuotas maxArrayLength="999999999" maxStringContentLength="9999999"/>
       <reliableSession enabled="true"/>
       <security mode="Message">
       <message clientCredentialType="Certificate"/>
       </security>
      </binding>
      </wsHttpBinding>
     </bindings>
    

    I am currently doing a cert for security but am not checking it since I only have a temp cert. Here is the sample code where I create the binding and set the endpoint and cert.

     Public Sub New(ByVal scopeName As String, ByVal url As String, ByVal FilterToken As String, ByVal certPath As String)
      Me.scopeName = scopeName
      Me.FilterToken = FilterToken
      Dim binding As New WSHttpBinding()
      binding.CloseTimeout = System.TimeSpan.FromSeconds(3000)
      binding.SendTimeout = System.TimeSpan.FromSeconds(3000)
      binding.ReceiveTimeout = System.TimeSpan.FromSeconds(3000)
      binding.OpenTimeout = System.TimeSpan.FromSeconds(3000)
    
      binding.ReaderQuotas.MaxArrayLength = 10485760
      binding.MaxReceivedMessageSize = 10485760
      binding.ReaderQuotas.MaxStringContentLength = 999999
      binding.Security.Message.ClientCredentialType = MessageCredentialType.Certificate
      binding.TransactionFlow = True
      binding.ReliableSession.Ordered = True
      binding.ReliableSession.Enabled = True
    
      Dim dns As New DnsEndpointIdentity("tempCert")
      Dim endPoint As New EndpointAddress(New Uri(url), dns)
      Dim cert As X509Certificate2 = New X509Certificate2(certPath, "", X509KeyStorageFlags.MachineKeySet)
    
      Dim factory As New ChannelFactory(Of IDynamicSyncContract)(binding, endPoint)
      factory.Credentials.ClientCertificate.Certificate = cert
      factory.Credentials.ServiceCertificate.Authentication.CertificateValidationMode = Security.X509CertificateValidationMode.None
      factory.Credentials.ServiceCertificate.Authentication.RevocationMode = X509RevocationMode.NoCheck
      proxy = factory.CreateChannel()
    
     End Sub
    
    • Edited by Rache Wednesday, March 9, 2011 7:58 PM added stack trace
    Wednesday, March 9, 2011 7:57 PM
  • Hi Rache,

    can you run SQL Profiler as well just to rule out if the query is getting to the server and if its the one taking some time to return?

    Thursday, March 10, 2011 7:38 AM
  • At this point in time the SQL has already been executed compiled and the data is being sent to the client. Either way the sql took about a second to execute since it's only returning 16k and change rows.

    My other test user has 550k rows which takes about 4-7 sec to return and it has no problems.

    My thoughts still come down to size. If I make the batch size 2000 which is greater than the size of the 16k rows it times out. If I set it below to 1000 it batches and works just fine.

    The time out is happening between the GetChanges function on the server and the GetBatchChanges on the proxy. The data has been gathered and it's just the return statement.

    Thursday, March 10, 2011 2:14 PM
  • Did you ever figure this out. I am having the same issue, using a proxy that is timing out at GetChangeBatch?
    Sunday, July 24, 2011 9:05 PM