locked
How to prevent multiple sync sessions when inactivityTimeout is exceeded? RRS feed

  • Question

  • --  After posting this msg, I realized a big error in my code, which I commented on below in a reply to my own post.  Please disregard this post.  I will repost with new questions and code, as apprpropriate.  Thanks.  --dan

    Hi,

    We are using Sync Framework 2.1 with SQL server db's on both ends.  This is in production, and works reasonably well, except when a sync takes too long.  

    Currently, inactivityTimeout is set for 01:56:00.  Occasionally, when a user does a large data import, a sync session exceeds this time limit.  Then we get a CommunicationException.  This causes the code to loop back and try again, starting another sync session.  Now there are two sessions trying to sync the same data, with one blocking the other, and both slowing down tremendously.  At this point, the only solution is either to wait many hours, or go into SQL Server and kill one of the spids.

    The code for the main method is below.  I feel like we are close, but I can't figure out how to recover properly from the CommunicationException.  I tried using the remote provider events to signal when the batch is done, but they do not seem to be firing.  

    A few questions:

    Is it possible to have a callback from the remote provider?

    Is there any downside to setting the inactivityTimeout to a very large value, such as 07:00:00?

    Here is the code (truncated a bit for readability, including omission of many of the catch blocks).

    Thanks.

    Dan

    public string DoSync(string strScopeName, bool bClearDirtyBit, out bool bDoSyncRequestToStop, out bool bSyncRetry)
    {
      RelationalSyncProvider localRelationalSyncProvider = null;
      RelationalSyncProvider remoteRelationalSyncProvider = null;
      int intExceptionCounter = 0;
      bDoSyncRequestToStop = false;
      bSyncRetry = false;
      bool bDbSyncException = false;
      
      while (true)
      {
        try
        {
          if (intExceptionCounter >= 10)
          {
            string strMessage = "DoSync - too many exceptions (" + intExceptionCounter.ToString() + ")";
            Helper.LoggerSingleThread.WriteLog(true, "code", strMessage, true, true);
            return strMessage;
          }
    
          if (!bSyncRetry)
          {
            localRelationalSyncProvider = Helper.SyncHelper.ConfigureLocalSqlSyncProvider(strScopeName, m_strLocalConnectionString, m_strCustomerGuid);
            localRelationalSyncProvider.ApplyChangeFailed += new EventHandler<DbApplyChangeFailedEventArgs>(LocalProvider_ApplyChangeFailed);
            localRelationalSyncProvider.SyncProgress += new EventHandler<DbSyncProgressEventArgs>(LocalProvider_SyncProgress);
            remoteRelationalSyncProvider = new Helper.SqlSyncProviderProxy(strScopeName, m_strCustomerGuid);
            remoteRelationalSyncProvider.SyncProgress += new EventHandler<DbSyncProgressEventArgs>(RemoteProvider_SyncProgress);
    
            remoteRelationalSyncProvider.BatchApplied += new EventHandler<DbBatchAppliedEventArgs>(RemoteProvider_BatchApplied);
            remoteRelationalSyncProvider.ChangesApplied += new EventHandler<DbChangesAppliedEventArgs>(RemoteProvider_ChangesApplied);
          }
    
          //  If it initially failed with DbSyncException, probably due to large field, so next try use a larger size limit
          if (!bDbSyncException)
          {
            localRelationalSyncProvider.MemoryDataCacheSize = Convert.ToUInt32((System.Configuration.ConfigurationManager.AppSettings["MemoryDataCacheSize"]));
            remoteRelationalSyncProvider.MemoryDataCacheSize = Convert.ToUInt32((System.Configuration.ConfigurationManager.AppSettings["MemoryDataCacheSize"]));
          }
          else
          {
            localRelationalSyncProvider.MemoryDataCacheSize = Convert.ToUInt32((System.Configuration.ConfigurationManager.AppSettings["MemoryDataCacheSizeRetry"]));
            remoteRelationalSyncProvider.MemoryDataCacheSize = Convert.ToUInt32((System.Configuration.ConfigurationManager.AppSettings["MemoryDataCacheSizeRetry"]));
          }
    
          Orchestrator = new SyncOrchestrator();
          Orchestrator.LocalProvider = localRelationalSyncProvider;
          Orchestrator.RemoteProvider = remoteRelationalSyncProvider;
    
          if (IsDataChanged(bClearDirtyBit) || bSyncRetry)
            Orchestrator.Direction = SyncDirectionOrder.UploadAndDownload;
          else
            Orchestrator.Direction = SyncDirectionOrder.Upload;
    
          SyncOperationStatistics syncStats = Orchestrator.Synchronize();
          
          bSyncRetry = false;
          bDbSyncException = false;
    
          SetIsSyncRunningFlag(false);	//  clear the flag
          
          String strSyncMsg = this.PrepareTheMessage(syncStats);
          Helper.LoggerSingleThread.LogEventLog(strSyncMsg, null, EventLogEntryType.Information, true);
          return strSyncMsg;
        }	
        catch (DbSyncException ex)
        {
          //  usually caused by large field size
          //  will loop around and try again, this time with the larger batch size
          Helper.LoggerSingleThread.LogEventLog(strScopeName + ": " + strLoggerSource + "DoSync DbSyncException.  Trying again ", ex, EventLogEntryType.Error);
          bSyncRetry = true;
          bDbSyncException = true;
        }
        catch (TimeoutException ex)
        {
          Helper.LoggerSingleThread.LogEventLog(strScopeName + ": " + strLoggerSource + "DoSync TimeoutException", ex, EventLogEntryType.Error);
          bSyncRetry = true;
        }
        catch (FaultException ex)
        {
          Helper.LoggerSingleThread.LogEventLog(strScopeName + ": " + strLoggerSource + "DoSync Fault Exception", ex, EventLogEntryType.Error);
          bSyncRetry = true;
          if (ex.Message.Contains("Conflict.Type: ErrorsOccurred"))	//  NB:  this string inserted in Welcome.RemoteProvider_ApplyChangeFailed
          {
            bDoSyncRequestToStop = true;
          }
        }
        catch (CommunicationException ex)
        {
          //  usually caused by inactivityTimeout
          Helper.LoggerSingleThread.LogEventLog(strScopeName + ": " + strLoggerSource + "DoSync CommunicationException", ex, EventLogEntryType.Error);
          bSyncRetry = true;
        }
        catch (ThreadAbortException ex)
        {
          Helper.LoggerSingleThread.LogEventLog(strScopeName + ": " + strLoggerSource + "DoSync ThreadAbortException", ex, EventLogEntryType.Error);
          bSyncRetry = true;
        }
        catch (ObjectDisposedException ex)
        {
          Helper.LoggerSingleThread.LogEventLog(strScopeName + ": " + strLoggerSource + "DoSync ObjectDisposedException", ex, EventLogEntryType.Error);
          bSyncRetry = true;
        }
        catch (Helper.SpecialException ex)
        {
          Helper.LoggerSingleThread.LogEventLog(strScopeName + ": " + strLoggerSource + "DoSync SpecialException", ex, EventLogEntryType.Error);
          bSyncRetry = true;
        }
        catch (Exception ex)
        {
          Helper.LoggerSingleThread.LogEventLog(strScopeName + ": " + strLoggerSource + "DoSync Exception", ex, EventLogEntryType.Error);
          bSyncRetry = true;
          if (ex.Message.Contains("Conflict.Type: ErrorsOccurred"))
          {
            bDoSyncRequestToStop = true;
          }
        }
        finally
        {
          try
          {
            if (bSyncRetry)
            {
              intExceptionCounter++;
            }
    
            if (!bSyncRetry)
            {
              if (localRelationalSyncProvider != null)
                ((IDisposable)localRelationalSyncProvider).Dispose();
    
              if (remoteRelationalSyncProvider != null)
                ((IDisposable)remoteRelationalSyncProvider).Dispose();
    
              if (Orchestrator != null)
                Orchestrator = null;
            }
          }
          catch (Exception ex)
          {
            throw;
          }
        }
    
        if (!bSyncRetry)
          return string.Empty;
    
      }	//  close while block
    }	// close DoSync



    Dan Hurwitz


    • Edited by Dan Hurwitz Tuesday, November 8, 2016 1:42 PM new knowledge
    Friday, November 4, 2016 9:16 PM

All replies

  • Hi,

    I just had a realization about something that is very wrong with the above code.  What happens is that if there is an exception where bSyncRetry is set true, which is all the catch blocks I've included in the code sample above, I do not dispose of the SyncProviders or Orchestrator in the finally block, but rather loop back and try to use the same Orchestrator to regain control of the session.  This is fundamentally flawed, because once the exception occurs, then the Orchestrator goes out of scope and is lost.  So I need to modify the code.

    The big question remains:  in the event of a CommunicationException, how do I retain control of the sync session which is still merrily cooking along on the WAN-side? (Ideally, I would like to at least be able to retrieve the sync stats.)  I know that is happening because I can see it using SQL Profiler on the remote DB server, and when it completes, I can verify that in fact the data has been synced.

    Thanks.

    Dan


    Dan Hurwitz

    Saturday, November 5, 2016 11:17 AM