none
Controlling timeouts when doing large sync RRS feed

  • Question

  • Hi,

    We are using the Sync Framework 2.1 with SQL Server 2012 on both ends.  It is in production, and works reasonably well.  One problem we have is with very large syncs which trigger some sort of timeout, resulting in a CommunicationException.   

    Originally, I thought that the controlling timeout was the inactivityTimeout, set in the LAN-side config file.  This was set to 00:01:56.  I bumped this up to 07:00:00, and ran another large test. It timed out after 2:02 with a CommunicationException.

    I then noticed both the sendTimeout and ReceiveTimeout were still set to 01:56:00.  So they were both changed to 07:00:00, again LAN-side.  Same test results.

    Then I made the same changes on the WAN-side config as well as the LAN-side, so both ends matched.  Same results.  

    In multiple tests, the CommunicationException occurs somewhere between 2:03 and  2:20 after the start of the session.

    We cannot find where this timeout period is coming from.  Any suggestions?

    Here are extracts from the two config files:

    LAN-side

    <system.serviceModel>
      <bindings>
        <customBinding>
          <binding name="largeMessageHttpBinding" closeTimeout="00:25:00" openTimeout="00:25:00" receiveTimeout="07:00:00" sendTimeout="07:00:00">
            <reliableSession ordered="true" maxPendingChannels="16384" maxRetryCount="100" inactivityTimeout="07:00:00"></reliableSession>
            <binaryMessageEncoding maxReadPoolSize="2147483647" maxWritePoolSize="2147483647" maxSessionSize="2147483647">
              <readerQuotas maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxDepth="64" maxNameTableCharCount="2147483647" maxStringContentLength="2147483647"></readerQuotas>
            </binaryMessageEncoding>
            <httpsTransport maxReceivedMessageSize="2147483647" maxBufferPoolSize="2147483647" maxBufferSize="2147483647"></httpsTransport>
          </binding>
        </customBinding>
      </bindings>
    </system.serviceModel>
    

    WAN-side

    <system.serviceModel>
      <behaviors>
        <serviceBehaviors>
          <behavior name="MyServiceBehavior">
            <dataContractSerializer maxItemsInObjectGraph="2147483647" />
            <serviceDebug includeExceptionDetailInFaults="true" />
            <serviceMetadata httpGetEnabled="true" />
            <serviceThrottling maxConcurrentCalls="2147483647" maxConcurrentInstances="2147483647" maxConcurrentSessions="2147483647"/>
          </behavior>
        </serviceBehaviors>
      </behaviors>
      <bindings>
        <customBinding>
          <binding name="HttpBinding"
            closeTimeout="00:25:00" openTimeout="00:25:00" receiveTimeout="07:00:00" sendTimeout="07:00:00" >
            <reliableSession ordered="true" maxPendingChannels="16384" maxRetryCount="100"  inactivityTimeout="07:00:00"/>
            <binaryMessageEncoding  maxReadPoolSize="2147483647" maxWritePoolSize="2147483647" maxSessionSize="2147483647">
              <readerQuotas maxArrayLength="2147483647" maxBytesPerRead="2147483647" maxDepth="64" maxNameTableCharCount="2147483647" maxStringContentLength="2147483647" />
            </binaryMessageEncoding>
            <httpTransport maxReceivedMessageSize="2147483647" maxBufferPoolSize="2147483647" maxBufferSize="2147483647"/>
          </binding>
        </customBinding>
      </bindings>
      <services>
        <service name="WanService.Welcome" behaviorConfiguration="MyServiceBehavior">
          <host>
            <baseAddresses>
              <add baseAddress="http://127.0.0.1:8082/WanService/Welcome" />
            </baseAddresses>
          </host>
          <endpoint address="" binding="customBinding" bindingConfiguration="HttpBinding" contract="Helper.IWelcome">
            <identity>
              <dns value="localhost" />
            </identity>
          </endpoint>
        </service>
      </services>
    </system.serviceModel>


    Thanks.

    --Dan


    Dan Hurwitz

    Tuesday, November 8, 2016 2:57 PM

Answers

  • I finally found the missing timeout values.  Not only must you set the receiveTimeout,  sendTimeout, and inactivityTimeout in both the LAN-side and WAN-side app.config files, but you must also set the same values in the WCF routing web.config.  In addition, we found that the SQL Server CommandTimeout must be set suitably large.  Here is an extract of all 3 config files.

    LAN-Side - app.config
    ---------
    <appSettings>
           <add key="ApplicationTransactionSize" value="10240" />
           <add key="CommandTimeout" value="1000" />
           <add key="MemoryDataCacheSize" value="5120" />
    </appSettings>
    <system.serviceModel>
           <bindings>
                  <customBinding>
                         <binding name="largeMessageHttpBinding" closeTimeout="00:25:00" openTimeout="00:25:00" receiveTimeout="07:00:00" sendTimeout="07:00:00" >
                               <reliableSession ordered="true" maxPendingChannels="16384" maxRetryCount="100" inactivityTimeout="07:00:00"/>
                         </binding>
                  </customBinding>
           </bindings>
    </system.serviceModel>
    
    WAN-side – app.config
    ----------------------
    <appSettings>
           <add key="ApplicationTransactionSize" value="10240" />
           <add key="CommandTimeout" value="1000" />
           <add key="MemoryDataCacheSize" value="5120" />
    </appSettings>
    <system.serviceModel>
           <bindings>
                  <customBinding>
                         <binding name="HttpBinding" closeTimeout="00:25:00" openTimeout="00:25:00" receiveTimeout="07:00:00" sendTimeout="07:00:00" >
                               <reliableSession ordered="true" maxPendingChannels="16384" maxRetryCount="100" inactivityTimeout="07:00:00"/>
                         </binding>
                  </customBinding>
           </bindings>
    </system.serviceModel>
    
    
    WAN-side – web.config
    ---------------------
    <system.web>
           <sessionState timeout="420" />
    </system.web>
    <system.serviceModel>
           <bindings>
                  <customBinding>
                         <binding name="HttpBinding" closeTimeout="00:25:00" openTimeout="00:25:00" receiveTimeout="07:00:00" sendTimeout="07:00:00" >
                               <reliableSession ordered="true" maxPendingChannels="16384" maxRetryCount="100" inactivityTimeout="07:00:00"/>
                         </binding>
                  </customBinding>
           </bindings>
    </system.serviceModel>
    
    


    Dan Hurwitz

    • Marked as answer by Dan Hurwitz Wednesday, November 23, 2016 3:39 PM
    Wednesday, November 23, 2016 3:39 PM

All replies

  • I should also mention that after this CommunicationException occurs, the sync continues to process on the WAN-side and completes successfully, although the SyncStats reported are incomplete.  I have verified however, that all the records are in fact synced up to the WAN.

    Thanks.

    --d


    Dan Hurwitz

    Tuesday, November 8, 2016 3:07 PM
  • Here is some follow-up information:

    Previously, I had set the following timeouts on both the client (LAN) and service (WAN) sides:

    • inactivityTimeout = 07:00:00
    • sendTimeout = 07:00:00
    • receiveTimeout = 07:00:00
    • openTimeout = 00:25:00
    • closeTimeout = 00:25:00

    In all the tests, it timed out between 2:03 and 2;20, as mentioned above.

    For this most recent test, I set the SqlSyncProvider.CommandTimeout to 6 hours (21600 seconds).  The time to failure increased to 3:25.  Once again, all the data was correctly synced, but the resultant CommunicationException prevented the return of SyncStats.


    Dan Hurwitz

    Wednesday, November 9, 2016 4:01 PM
  • I finally found the missing timeout values.  Not only must you set the receiveTimeout,  sendTimeout, and inactivityTimeout in both the LAN-side and WAN-side app.config files, but you must also set the same values in the WCF routing web.config.  In addition, we found that the SQL Server CommandTimeout must be set suitably large.  Here is an extract of all 3 config files.

    LAN-Side - app.config
    ---------
    <appSettings>
           <add key="ApplicationTransactionSize" value="10240" />
           <add key="CommandTimeout" value="1000" />
           <add key="MemoryDataCacheSize" value="5120" />
    </appSettings>
    <system.serviceModel>
           <bindings>
                  <customBinding>
                         <binding name="largeMessageHttpBinding" closeTimeout="00:25:00" openTimeout="00:25:00" receiveTimeout="07:00:00" sendTimeout="07:00:00" >
                               <reliableSession ordered="true" maxPendingChannels="16384" maxRetryCount="100" inactivityTimeout="07:00:00"/>
                         </binding>
                  </customBinding>
           </bindings>
    </system.serviceModel>
    
    WAN-side – app.config
    ----------------------
    <appSettings>
           <add key="ApplicationTransactionSize" value="10240" />
           <add key="CommandTimeout" value="1000" />
           <add key="MemoryDataCacheSize" value="5120" />
    </appSettings>
    <system.serviceModel>
           <bindings>
                  <customBinding>
                         <binding name="HttpBinding" closeTimeout="00:25:00" openTimeout="00:25:00" receiveTimeout="07:00:00" sendTimeout="07:00:00" >
                               <reliableSession ordered="true" maxPendingChannels="16384" maxRetryCount="100" inactivityTimeout="07:00:00"/>
                         </binding>
                  </customBinding>
           </bindings>
    </system.serviceModel>
    
    
    WAN-side – web.config
    ---------------------
    <system.web>
           <sessionState timeout="420" />
    </system.web>
    <system.serviceModel>
           <bindings>
                  <customBinding>
                         <binding name="HttpBinding" closeTimeout="00:25:00" openTimeout="00:25:00" receiveTimeout="07:00:00" sendTimeout="07:00:00" >
                               <reliableSession ordered="true" maxPendingChannels="16384" maxRetryCount="100" inactivityTimeout="07:00:00"/>
                         </binding>
                  </customBinding>
           </bindings>
    </system.serviceModel>
    
    


    Dan Hurwitz

    • Marked as answer by Dan Hurwitz Wednesday, November 23, 2016 3:39 PM
    Wednesday, November 23, 2016 3:39 PM