After SyncAbortedException Synchronize() takes a very long time
-
Wednesday, October 27, 2010 6:15 PM
Using the MS Sync Framework, I have a SyncOrchestrator and the two file FileSyncProviders. During a normal Synchronize() call, it takes about 3 minutes to finish the synchronize process.
After a SyncAbortedException, the same process takes 45 minutes.
How can I clean up after a SyncAbortedException?
Why does it take so long after a SyncAbortedException?
Thanks
Jim
All Replies
-
Wednesday, October 27, 2010 7:53 PMThat does seem a very long time. Have you tried running any profiling tools (Visual Studio or Red Gate ANTS Performance Profiler) to see exactly which code is taking so long to run? Which version of the Sync Framework are you using?
-
Wednesday, October 27, 2010 8:17 PM
I'm using the Sync Framework v2.0.
The function that is taking so long is SyncOrchestrator.Synchronize(). My profiler will not give me further details.
It only takes a long time after I cancel the Synchronize operation or after I lose connection to my remote data path (ie network is unreachable).
-
Friday, October 29, 2010 4:40 PM
What does the Sync Framework 2.0/2.1 expect the programmer to do after an aborted synchronize()?
More information....
1) First synchronize (4-5 minutes)
2) Next synchronize with no file changes (2-3 minutes)
3) Start another synchronize and abort it (call SyncOrchestrator.Cancel())
a) Throws SyncAbortedExctption (expected)
4) Next synchronze (10-50 minutes)
This results in lots of uploads, downloads or both even though
the filesystem has not changed.
Immedetely After this synchronize()...
a) i. Clean up tombstones (metadata.CleanupDeletedItems())
ii. Next synchronize -> throws NotImplementedException
b) i. DON'T Clean up tombstones
ii. Next synchronize -> OK
Normal Full Sync - Initial (No Metadata)
-------------------------------------------------------------------------------
Sync Start Time: 10/28/2010 10:37:45 AM
Download Applied: 31078
Download Failed: 0
Download Total: 31078
Upload Applied: 23432
Upload Failed: 0
Upload Total: 23432
Sync End Time: 10/28/2010 10:42:02 AM
Normal Full Sync - No Changes (Using current Metadata)
-------------------------------------------------------------------------------
Sync Start Time: 10/28/2010 11:37:45 AM
Download Applied: 0
Download Failed: 0
Download Total: 0
Upload Applied: 0
Upload Failed: 0
Upload Total: 0
Sync End Time: 10/28/2010 11:39:22 AM
After SyncAbortedException - No Changes in FileSystem (Using current Metadta)
--Run A------------------------------------------------------------------------
Sync Start Time: 10/28/2010 8:50:05 AM
Download Applied: 3962
Download Failed: 0
Download Total: 3962
Upload Applied: 0
Upload Failed: 0
Upload Total: 0
Sync End Time: 10/28/2010 9:07:59 AM
After SyncAbortedException - No Changes in FileSystem (Using current Metadta)
--Run B-----------------------------------------------------------------------
Sync Start Time: 10/29/2010 8:34:10 AM
Download Applied: 1
Download Failed: 0
Download Total: 1
Upload Applied: 7419
Upload Failed: 1
Upload Total: 7420
Sync End Time: 10/29/2010 9:01:29 AM
If I clean up tombstone items (ie CleanupDeletedItems()) after this step, the
next synchronize() will throw NotImplementedException. -
Friday, October 29, 2010 8:47 PM
Even more info....
This time it took 90 minutes (After a network failure). And for most of that 90 minutes it was using 100% of one of my two CPUs.
It's obvious, by the lack of responses to this thread, that no one is using MS Sync Framework (FileSyncProvider & SyncOrchestrator) for serious synchronization. It is still a beta product that can only be used as a Sync Toy.
1) First synchronize (4-5 minutes)
-------------------------------------------------------------------------------
2) Next synchronize with no file changes (2-3 minutes)
3) Start another synchronize (between local and network share). Then, pull the network cable.
a) (expected) Throws SyncException: The FileSyncProvider failed while detecting changes.
4) Next synchronze (90 minutes)!!!!Sync Start Time: 10/29/2010 11:30:27 AM
Download Applied: 8170
Download Failed: 0
Download Total: 8170
Upload Applied: 0
Upload Failed: 0
Upload Total: 0
Sync End Time: 10/29/2010 1:00:11 PM
-
Saturday, October 30, 2010 3:54 AMModerator
Hi,
May you provide more information for me to understand your scenario better?
1. For your first sync, it only took 4-5 mins to sync totally 31078 + 23432 = 54510 files and folders. It is a lot of files. May I know what is the average size of your files? They should be pretty small files, right?
2. Without sync, how fast for you to copy these files to the same remote share? Since your empty sync takes about 2-3 minutes that mainly change detection times, I assume the full copy time is less than 2 minutes for your file system. Can you verify if it is correct?
3. After SyncAbortException, your next time took a lot time, but it is not an empty sync. They are thousands of changes synced in either download or upload syncs. May I know how many files are inserted/updated/deleted per hour on your file system? As you know, FileSyncProvider metadata also track your delete files/folders. I would lilke to guess how many deletes accumulated on your file system when you first hit a SyncAbortException.
4. May you retry your above scenario by calling FileSyncProvider.DetectChanges for both providers before SyncOrchestrator.Sync() and measure how long the ChangeDetects() calls takes? You need to create your FileSyncProvider with FileSyncOptions.ExplicitDetectChanges in this case.
5. Just want to confirm, after SyncAbortException, you didn't reuse the same FileSyncProvider instances to start a new sync, right?
6. Last, because your initial sync didn't take very long, After SyncAbortException, have you tried to delete the .metadata file from both root folders before re-sync? If it can make the sync speed to be normal again, there might be some reasons to explain it.
Thanks,
Dong
This posting is provided AS IS with no warranties, and confers no rights. -
Monday, November 01, 2010 4:36 PM
1) Yes, the files are small and there may actually be more directories than files.
2) The total copy time is about 10-15 minutes. The total sync time, when the local filesystem is empty, is about the same 10-15 minutes.
3) After SyncAbortedException, I did not change any files on the local or remote filesystem. It appears the metadata is corrupted or confused.
This is a little complicated but shouldn't be hard to follow.
a. Using the last successful sync metadata after SyncAbortedException results in a quick normal sync.
i) Successful sync -> save metadata (this is a good copy of the metadata)
ii) Start sync, then, Abort -> delete metadata and replace with saved metadata from above (the good copy from a successful sync)
iii) Start sync, results in a normal 1-3 minute sync process
No files were changed on the filesystem. The only thing different between case (a.) and case (b.) is the metadata.
b. Using the same metadata after SyncAbortedException results in long sync
i) Successful sync
ii) Start sync, then, Abort (do NOT delete or modify metadata)
iii) Start sync, results in a very long 15-90 minute sync
From my testing, whenever SyncOrchestrator.Synchronize() throws an exception, the metadata is corrupted or confused.
The following exceptions cause the next sync to take a very long time.
SyncAbortedExceptio, InvalidOperationException, SyncException (I have not yet tested FileNotFoundException)
Sometimes the metadata is so corrupted the next sync results in NotImplementedException and the metadata must be deleted.
4) Looks like most of the time is spent in SyncOrchestrator.Synchronize()
Normal Synchronization
LocalFileSyncProvider.DetectChanges: Time in seconds:16
RemoteFileSyncProvider.DetectChanges: Time in seconds:36
----------------------------------------------
Sync Start Time: 11/1/2010 1:37:17 PM
Download Applied: 0
Download Failed: 0
Download Total: 0
Upload Applied: 0
Upload Failed: 0
Upload Total: 0
Sync End Time: 11/1/2010 1:37:20 PM
----------------------------------------------
SyncTime: 16+36+3 = 55 seconds
After SyncAbortedException
LocalFileSyncProvider.DetectChanges: Time in seconds:24
RemoteFileSyncProvider.DetectChanges: Time in seconds:65
----------------------------------------------
Sync Start Time: 11/1/2010 1:46:54 PM
Download Applied: 8556
Download Failed: 0
Download Total: 8556
Upload Applied: 0
Upload Failed: 0
Upload Total: 0
Sync End Time: 11/1/2010 3:15:19 PM
----------------------------------------------
SyncTime: 24+65+3 = 5305 seconds = 1 hr 29 min 54 sec
5) I've tried it both ways, reusing the FileSyncProvider and not reusing it. Also, I have completely stopped my application after the SyncAbortedException and restarted the application. The results are the same for all cases, a long sync time.
6) This is well explained in number 3).
Why would I want to delete my metadata files? The sync framework cannot properly detect changes without the metadata. That completely defeats the purpose of having the metadata to begin with.
Thanks
Jim -
Monday, November 01, 2010 10:31 PMJust to let you know I updated 4).
-
Tuesday, November 02, 2010 7:06 AMModerator
Hi,
I tried to write a sync app with FileSyncProvider to repro your scenario, but I cannot.
I first did a bi-direction normal sync to bring bith folders in sync. After calling SyncOrchestrator.Cancel() in the first ApplyingChangeEvent of the second sync, the Sync session is stopped by SyncAbortException. When I tried to sync again immediately, I saw a normal empty sync that finished with 2-3 seconds. I have created enough items and one provider's change detection time is around 17 seconds.
May you share the exact source code for repro?
Thanks,
Dong
This posting is provided AS IS with no warranties, and confers no rights. -
Tuesday, November 02, 2010 3:18 PM
I'll see what I can drum up.
Thanks, Jim
-
Tuesday, November 02, 2010 6:02 PM
Here are my test results using a test project. This test project uses the same code for the sync providers and sync orchestrator as my original project. Moreover, I get the same results.
I can give you the entire test project if you have a place to share files. Until you or I come up with a place to share files, I will post the entire content of the classes here.
This post only contains data from my test. The next post will contain the code.
Size: 559 MB (587,043,458 bytes)
Size on disk: 570 MB (598,253,568 bytes)
Contains: 7,771 Files, 15,336 Folders
1) Synchronize: Executing Synchronize() -> Complete (2 min 18 sec)
2) Synchronize: Executing Synchronize() -> Cancel
3) Synchronize: Executing Synchronize() -> Complete (2 min 18 sec)
4) Synchronize: Executing Synchronize() -> Cancel
5) Synchronize: Executing Synchronize() -> Complete (5 min 5 sec)
6) Synchronize: Executing Synchronize() -> Cancel
7) Synchronize: Executing Synchronize() -> Complete (22 min 10 sec)
Synchronize: Executing Synchronize() Function
Synchronize: Synchronize() Complete
Synchronize Status
Sync Start Time: 11/2/2010 9:48:46 AM
Download Applied: 0
Download Failed: 0
Download Total: 0
Upload Applied: 0
Upload Failed: 0
Upload Total: 0
Sync End Time: 11/2/2010 9:51:04 AM
The thread 0x131c has exited with code 0 (0x0).
Synchronize: Executing Synchronize() Function
Cancel: SyncOrchestrator Cancelling...
A first chance exception of type 'Microsoft.Synchronization.SyncAbortedException' occurred in Microsoft.Synchronization.dll
The thread 0x17f8 has exited with code 0 (0x0).
Synchronize: Executing Synchronize() Function
Synchronize: Synchronize() Complete
Synchronize Status
Sync Start Time: 11/2/2010 9:51:35 AM
Download Applied: 0
Download Failed: 0
Download Total: 0
Upload Applied: 0
Upload Failed: 0
Upload Total: 0
Sync End Time: 11/2/2010 9:53:53 AM
The thread 0x14d0 has exited with code 0 (0x0).
Synchronize: Executing Synchronize() Function
Cancel: SyncOrchestrator Cancelling...
A first chance exception of type 'Microsoft.Synchronization.SyncAbortedException' occurred in Microsoft.Synchronization.dll
The thread 0xa58 has exited with code 0 (0x0).
Synchronize: Executing Synchronize() Function
The thread 0x1740 has exited with code 0 (0x0).
Synchronize: Synchronize() Complete
Synchronize Status
Sync Start Time: 11/2/2010 9:59:43 AM
Download Applied: 1794
Download Failed: 0
Download Total: 1794
Upload Applied: 0
Upload Failed: 0
Upload Total: 0
Sync End Time: 11/2/2010 10:04:48 AM
The thread 0x968 has exited with code 0 (0x0).
Synchronize: Executing Synchronize() Function
Cancel: SyncOrchestrator Cancelling...
A first chance exception of type 'Microsoft.Synchronization.SyncAbortedException' occurred in Microsoft.Synchronization.dll
The thread 0x15d4 has exited with code 0 (0x0).
Synchronize: Executing Synchronize() Function
The thread 0x111c has exited with code 0 (0x0).
Synchronize: Synchronize() Complete
Synchronize Status
Sync Start Time: 11/2/2010 10:14:31 AM
Download Applied: 5740
Download Failed: 0
Download Total: 5740
Upload Applied: 0
Upload Failed: 0
Upload Total: 0
Sync End Time: 11/2/2010 10:36:41 AM
The thread 0xd2c has exited with code 0 (0x0).
-
Tuesday, November 02, 2010 6:13 PM
The Form1 class is a basic form for testing
-------------------------------------------------------------------- | | | -------------------------------------------------------- | | | local directory TextBox1 | | | -------------------------------------------------------- | | -------------------------------------------------------- | | | remote directory TextBox2 | | | -------------------------------------------------------- | | | | --------------------- --------------------- | | | sync Button1 | | cancel Button2 | | | --------------------- --------------------- | | | | -------------------------------------------------------- | | | output display TextBox3 | | | | | | | | | | | | | | | | | | | | | | | | | | | -------------------------------------------------------- | | | --------------------------------------------------------------------
Public Class Form1 Private m_FileSynchronizer As FileSyncFileGroup = Nothing Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click If m_FileSynchronizer Is Nothing Then m_FileSynchronizer = New FileSyncFileGroup(SyncDirectionOrder.UploadAndDownload, TextBox1.Text, TextBox2.Text) AddHandler m_FileSynchronizer.SynchronizeComplete, AddressOf SyncProcessCompleted TextBox1.Enabled = False TextBox2.Enabled = False End If m_FileSynchronizer.StartSynchProcess() End Sub Private Delegate Sub SyncProcessCompletedDelegate(ByVal syncResult As String) Private Sub SyncProcessCompleted(ByVal syncResult As String) Dim i As System.ComponentModel.ISynchronizeInvoke = _ CType(TextBox3, System.ComponentModel.ISynchronizeInvoke) If i.InvokeRequired Then Dim tempDelegate As New SyncProcessCompletedDelegate(AddressOf SyncProcessCompleted) Dim args() As Object = {syncResult} i.BeginInvoke(tempDelegate, args) Return End If TextBox3.Text = TextBox3.Text + vbCrLf + syncResult End Sub Private Sub Button2_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button2.Click If m_FileSynchronizer IsNot Nothing Then m_FileSynchronizer.CancelSynchProcess() End If End Sub End Class Public Class FileSyncFileGroup Public Event SynchronizeComplete(ByVal syncResult As String) Private m_SyncDirection As SyncDirectionOrder = SyncDirectionOrder.UploadAndDownload Private m_LocalDir As String = String.Empty Private m_RemoteDir As String = String.Empty Private m_SyncOrchestrator As SyncOrch = Nothing Private m_NeedsFullSynchronization As Boolean = False Private m_WorkerThread As BackgroundWorker = Nothing Private m_SynchronizeThread As Thread = Nothing Private Sub New() ' Disallow Default Constructor End Sub Public Sub New( _ ByVal nDirection As SyncDirectionOrder, _ ByVal nLocalPath As String, _ ByVal nRemotePath As String) MyBase.New() Dim mLocalDir As DirectoryInfo = Nothing Dim mRemoteDir As DirectoryInfo = Nothing m_SyncDirection = nDirection m_LocalDir = nLocalPath mLocalDir = New DirectoryInfo(nLocalPath) If mLocalDir Is Nothing Then Throw New DirectoryNotFoundException("Local Path is invalid: " + nLocalPath) ElseIf Not mLocalDir.Exists Then Directory.CreateDirectory(mLocalDir.FullName) End If m_RemoteDir = nRemotePath mRemoteDir = New DirectoryInfo(nRemotePath) If mRemoteDir Is Nothing Then Throw New DirectoryNotFoundException("Remote Path is invalid: " + nLocalPath) ElseIf Not mRemoteDir.Exists Then ' Let's not do this.... ' The network might be down or temporally off line when we start 'Throw New DirectoryNotFoundException("Remote Path does not exist: " + nLocalPath) End If m_SyncOrchestrator = New SyncOrch(Me) m_WorkerThread = New BackgroundWorker() m_WorkerThread.WorkerSupportsCancellation = True AddHandler m_WorkerThread.DoWork, AddressOf MonitorFullSyncStatus m_WorkerThread.RunWorkerAsync() End Sub Private Function CheckLocalPathIsAvailable() As Boolean Return Directory.Exists(m_LocalDir) End Function Private Function CheckRemotePathIsAvailable() As Boolean Return Directory.Exists(m_RemoteDir) End Function Private Function BothPathsAvailable() As Boolean Return (CheckRemotePathIsAvailable() And CheckLocalPathIsAvailable()) End Function Private Sub MonitorFullSyncStatus(ByVal sender As System.Object, ByVal e As System.ComponentModel.DoWorkEventArgs) Dim worker As BackgroundWorker = CType(sender, BackgroundWorker) m_NeedsFullSynchronization = False While Not worker.CancellationPending Try If (m_NeedsFullSynchronization AndAlso BothPathsAvailable()) Then If (Not SynchronizeThreadIsAlive()) Then StartSynchProcess() End If ElseIf (Not BothPathsAvailable()) Then ' Make sure the sync process is not running while the network path is not available. ' The sync process tends to get into an infinate loop when it loses the network path. CancelSynchProcess() End If Catch ex As Exception Console.WriteLine("MonitorFullSyncStatus: Exception: " + ex.ToString) End Try Thread.Sleep(10000) End While End Sub Private Sub Synchronize() Dim syncResult As String Try syncResult = m_SyncOrchestrator.Synchronize() If (syncResult.Contains("Success:")) Then m_NeedsFullSynchronization = False End If Catch ex As ThreadInterruptedException syncResult = "ERROR: The File Sync operation has been canceled." Catch ex As Exception syncResult = "ERROR: Exception: The File Sync operation has thrown an exception." + vbNewLine + ex.ToString End Try RaiseEvent SynchronizeComplete(syncResult) End Sub Private Function SynchronizeThreadIsAlive() As Boolean SynchronizeThreadIsAlive = False If (m_SynchronizeThread IsNot Nothing) Then SynchronizeThreadIsAlive = m_SynchronizeThread.IsAlive End If End Function Public Sub StartSynchProcess() If (SynchronizeThreadIsAlive()) Then Exit Sub End If m_SynchronizeThread = New Thread(AddressOf Synchronize) m_SynchronizeThread.Priority = ThreadPriority.Lowest m_SynchronizeThread.Start() End Sub Public Sub CancelSynchProcess() m_SyncOrchestrator.Cancel() ' Wait a few seconds for the thread to die Dim tenSecTimer As New Stopwatch() tenSecTimer.Start() While (SynchronizeThreadIsAlive()) If tenSecTimer.ElapsedMilliseconds > 10000 Then tenSecTimer.Stop() tenSecTimer = Nothing Exit While End If Thread.Sleep(100) End While End Sub Public ReadOnly Property OrchestratorSyncDirection() As SyncDirectionOrder Get Return m_SyncDirection End Get End Property Public ReadOnly Property LocalDir() As String Get Return m_LocalDir End Get End Property Public ReadOnly Property RemoteDir() As String Get Return m_RemoteDir End Get End Property Public ReadOnly Property SyncOptions() As FileSyncOptions Get SyncOptions = _ FileSyncOptions.RecycleConflictLoserFiles Or _ FileSyncOptions.RecycleDeletedFiles Or _ FileSyncOptions.RecyclePreviousFileOnUpdates End Get End Property Public ReadOnly Property SyncState() As SyncOrchestratorState Get Dim retSyncState As SyncOrchestratorState retSyncState = m_SyncOrchestrator.SyncState If (retSyncState = SyncOrchestratorState.Ready) Or (retSyncState = SyncOrchestratorState.Canceled) Then If (SynchronizeThreadIsAlive()) Then retSyncState = SyncOrchestratorState.UploadingAndDownloading End If End If Return retSyncState End Get End Property Public Property NeedsFullSynchronization() As Boolean Get Return m_NeedsFullSynchronization End Get Set(ByVal value As Boolean) m_NeedsFullSynchronization = value End Set End Property End Class Imports System.Reflection Public Class SyncOrch Private Shared m_MetadataStorageBasePath As String = String.Empty Private m_LocalFileSyncProvider As FileSyncProvider = Nothing Private m_RemoteFileSyncProvider As FileSyncProvider = Nothing Private m_SyncOrchestrator As SyncOrchestrator = Nothing Private m_LastSuccessfulSyncTime As Date = Nothing Private m_SyncGroup As FileSyncFileGroup = Nothing Private m_ScopeFilter As FileSyncScopeFilter = Nothing Private Sub New() ' Disallow Default Constructor End Sub Public Sub New(ByRef nSyncGroup As FileSyncFileGroup) MyBase.New() Try m_SyncGroup = nSyncGroup m_SyncOrchestrator = New SyncOrchestrator() m_SyncOrchestrator.Direction = m_SyncGroup.OrchestratorSyncDirection Catch ex As Exception Console.WriteLine("New: Exception: " + ex.ToString) End Try End Sub Public Function Synchronize() As String Dim stats As Microsoft.Synchronization.SyncOperationStatistics = Nothing Dim syncResult As String = "ERROR: Sync Not Started for FileGroup " Try If (m_SyncOrchestrator.State = SyncOrchestratorState.Ready) Or (m_SyncOrchestrator.State = SyncOrchestratorState.Canceled) Then Console.WriteLine("Synchronize: Executing Synchronize() Function") m_SyncOrchestrator.Direction = m_SyncGroup.OrchestratorSyncDirection stats = SyncOrchestrator.Synchronize() syncResult = "Success: " m_LastSuccessfulSyncTime = Now Console.WriteLine("Synchronize: Synchronize() Complete") Else Console.WriteLine("Synchronize: SyncOrchestrator is busy.... NOT Executing Synchronize() Function") syncResult = "In-Process: Sync Busy " End If Catch ioex As InvalidOperationException ' Thrown by SyncOrchestrator if State is Not Ready and Not Canceled syncResult = "In-Process(Operation-based): " Catch saex As SyncAbortedException ' Thrown by SyncOrchestrator on call to Synchronize when the ' FileSyncProvider operation has been canceled.. syncResult = "ERROR: SyncAbortedException: The FileSyncProvider operation has been canceled." Catch fnfex As FileNotFoundException ' Thrown by SyncOrchestrator on call to Synchronize when the ' Local or Remote folder is no longer available. Probable cause is network disconnect. syncResult = "ERROR: FileNotFoundException: The FileSyncProvider failed while detecting changes." + _ vbNewLine + "Probable cause is RemoteDir <" + m_SyncGroup.RemoteDir + "> is inaccessible." Catch syncex As SyncException ' Thrown by SyncOrchestrator on call to Synchronize when the ' Local or Remote folder is no longer available. Probable cause is network disconnect. If (syncex.Message.Contains("The FileSyncProvider failed while detecting changes.")) Then syncResult = "ERROR: SyncException: The FileSyncProvider failed while detecting changes." + _ vbNewLine + "Probable cause is RemoteDir <" + m_SyncGroup.RemoteDir + "> is inaccessible due to network disconnect." Else syncResult = "ERROR: SyncException: " + syncex.ToString End If Catch ex As NotImplementedException syncResult = "ERROR: NotImplementedException: The File Sync operation has tried to execute a function that was not implemented." Catch otherex As Exception syncResult = "ERROR: Exception: " + otherex.ToString Finally If stats IsNot Nothing Then ' Display the SyncOperationStatistics Dim statusString As String = _ vbNewLine & _ "Sync Start Time: " & stats.SyncStartTime & _ vbNewLine & _ "Download Applied: " & stats.DownloadChangesApplied & _ vbNewLine & _ "Download Failed: " & stats.DownloadChangesFailed & _ vbNewLine & _ "Download Total: " & stats.DownloadChangesTotal & _ vbNewLine & _ "Upload Applied: " & stats.UploadChangesApplied & _ vbNewLine & _ "Upload Failed: " & stats.UploadChangesFailed & _ vbNewLine & _ "Upload Total: " & stats.UploadChangesTotal & _ vbNewLine & _ "Sync End Time: " & stats.SyncEndTime Console.WriteLine("Synchronize Status " + statusString) End If End Try Return syncResult End Function Public Sub Cancel() Try Console.WriteLine("Cancel: SyncOrchestrator Cancelling...") If (m_SyncOrchestrator IsNot Nothing) _ AndAlso ((m_SyncOrchestrator.State > SyncOrchestratorState.Ready) _ And (m_SyncOrchestrator.State < SyncOrchestratorState.Canceling)) Then m_SyncOrchestrator.Cancel() End If Catch ex As Exception ' NOTE: May Throw System.InvalidOperationException: ' This operation is not valid in any state of the object ' except for SyncAgentState.Uploading or ' SyncAgentState.Downloading. - IGNORE Console.WriteLine("Cancel: Threw exception (IGNORE) " + ex.ToString) End Try End Sub Private Function GetLocalMetadataFileName() As String Return "local.metadata" End Function Private Function GetRemoteMetadataFileName() As String Return "remote.metadata" End Function Private Function GetLocalTempPath() As String Dim result As String result = MetadataStoragePath() + "local.temp" If Not Directory.Exists(result) Then Directory.CreateDirectory(result) End If Return result End Function Private Function GetRemoteTempPath() As String Dim result As String result = MetadataStoragePath() + "remote.temp" If Not Directory.Exists(result) Then Directory.CreateDirectory(result) End If Return result End Function Private Function GetLocalConflictLoserPath() As String Dim result As String result = MetadataStoragePath() + "local.conflict" If Not Directory.Exists(result) Then Directory.CreateDirectory(result) End If Return result End Function Private Function GetRemoteConflictLoserPath() As String Dim result As String result = MetadataStoragePath() + "remote.conflict" If Not Directory.Exists(result) Then Directory.CreateDirectory(result) End If Return result End Function Private Function MetadataStoragePath() As String Dim result As String ' TBD: Should this be a config parameter? If (String.IsNullOrEmpty(m_MetadataStorageBasePath)) Then m_MetadataStorageBasePath = Path.GetDirectoryName(Assembly.GetEntryAssembly.Location) End If result = Path.Combine(m_MetadataStorageBasePath, "MEIFileSyncServiceMetadata\" + "\") If Not Directory.Exists(result) Then Directory.CreateDirectory(result) End If Return result End Function Public ReadOnly Property LastSuccessfullSyncTime() As Date Get Return m_LastSuccessfulSyncTime End Get End Property Public ReadOnly Property SyncState() As SyncOrchestratorState Get If (m_SyncOrchestrator IsNot Nothing) Then Return m_SyncOrchestrator.State Else Return SyncOrchestratorState.Canceled End If End Get End Property Private ReadOnly Property ScopeFilter() As FileSyncScopeFilter Get If m_ScopeFilter Is Nothing Then m_ScopeFilter = New FileSyncScopeFilter() m_ScopeFilter.AttributeExcludeMask = Nothing m_ScopeFilter.FileNameIncludes.Clear() m_ScopeFilter.FileNameExcludes.Clear() m_ScopeFilter.SubdirectoryExcludes.Clear() End If Return m_ScopeFilter End Get End Property Private ReadOnly Property LocalFileSyncProvider() As FileSyncProvider Get If m_LocalFileSyncProvider Is Nothing Then Try m_LocalFileSyncProvider = New FileSyncProvider( _ m_SyncGroup.LocalDir, _ Me.ScopeFilter, _ m_SyncGroup.SyncOptions, _ Me.MetadataStoragePath, _ GetLocalMetadataFileName(), _ GetLocalTempPath(), _ Nothing) Catch exDir As DirectoryNotFoundException ' This exception is expected when the local directory is not available Console.WriteLine("LocalFileSyncProvider: Could not create Local Provider because Local Directory not available: " + vbCrLf + exDir.ToString) m_LocalFileSyncProvider = Nothing Catch ex As Exception Console.WriteLine("LocalFileSyncProvider: Could not create Local Provider: " + vbCrLf + ex.ToString) m_LocalFileSyncProvider = Nothing End Try End If Return m_LocalFileSyncProvider End Get End Property Private ReadOnly Property RemoteFileSyncProvider() As FileSyncProvider Get If m_RemoteFileSyncProvider Is Nothing Then Try m_RemoteFileSyncProvider = New FileSyncProvider( _ m_SyncGroup.RemoteDir, _ Me.ScopeFilter, _ m_SyncGroup.SyncOptions, _ Me.MetadataStoragePath, _ GetRemoteMetadataFileName(), _ GetRemoteTempPath(), _ Nothing) Catch exDir As DirectoryNotFoundException ' This exception is expected when the local directory is not available Console.WriteLine("RemoteFileSyncProvider: Could not create Remote Provider because Remote Directory not available: " + vbCrLf + exDir.ToString) m_RemoteFileSyncProvider = Nothing Catch ex As Exception Console.WriteLine("RemoteFileSyncProvider: Could not create Remote Provider: " + vbCrLf + ex.ToString) m_RemoteFileSyncProvider = Nothing End Try End If Return m_RemoteFileSyncProvider End Get End Property Private ReadOnly Property SyncOrchestrator() As SyncOrchestrator Get ' This is here because sometimes the network is not available ' or the network path is not available during startup. ' When this happens, the remote path will not be accessable ' and the RemoteFileSyncProvider will be Nothing. ' Using the properties and assigning them each time will attempt ' to create the RemoteFileSyncProvider in the case where it ' is Nothing. If RemoteFileSyncProvider is valid, it just ' returns the m_RemoteFileSyncProvider class variable. m_SyncOrchestrator.LocalProvider = LocalFileSyncProvider() m_SyncOrchestrator.RemoteProvider = RemoteFileSyncProvider() Return m_SyncOrchestrator End Get End Property End Class
-
Wednesday, November 03, 2010 1:40 AMModerator
Hi,
Thanks for sharing the implemetation, but I may not be able to create a repro with your code because it looks like a combination of multiple partial files. It is difficult to create a project with them.
If you can confirm below points in your implementation, I can try to repro again:
1. Sync is started in a separate thread, and Cancel() is called from main thread after some wait time.
2. SyncOrchestrator instance is re-created for every sync
3. SyncOptions is RecycleConflitLoserFiles & RecyclDeletedFiles & RecyclePreviousFileOnUpdates. For remote share, recycle bin is not available. Did you set the same SyncOptions for remote provider?
4. Metadata Folder for both providers are in local directory
5. Need multiple retries to hit one repro, or repro is intermittent.
I have two more quesitons for you:
1. When hit a repro, did every sync after the SyncAbortException take very long time or just the first one?
2. You didn't explicitly call ChangeDetection method and didn't disable provider implicit change detection. It means the cancel() may happen in the middle of change detection, is it correct understanding?
Thanks,
Dong
This posting is provided AS IS with no warranties, and confers no rights. -
Wednesday, November 03, 2010 3:54 PM
Yes the project is 3 files; 1 form and 2 classes. Each class is its own file. I tried to draw the layout of the form but it did not come through very clearly.
I tried to reproduce, in a temporary project, the same code that my full project is using.
1. Yes, Sync is started in a separate thread and the cancel is triggered from a button click in the UI.
2. SyncOrchestrator is NOT recreated each time. My testing has confirmed that this is irrelevant. Using the same SyncOrchestrator or creating a new one does not make any difference as far as this issues is concerned.
3. SyncOptions = FileSyncOptions.RecycleConflictLoserFiles Or FileSyncOptions.RecycleDeletedFiles Or FileSyncOptions.RecyclePreviousFileOnUpdates
This is true for both the local and remote provider.
4. Yes, metadata is stored locally for both providers.
5. True. In some cases I have reproduced the problem in one try. I provided my test data "as is". I have no explanation for why it took multiple attempts to repro.
Questions:
1. Just the first one. After the long sync time, things return normal.
2. I removed the explicit change detection code. However, the cancel is most likely happening during the change detection. If I remember correctly, using explicit change detection, the cancel happened during the DetectChanges() function. This resulted in a SyncAbortedExctption. On the following sync attempt, the DetectChanges happened very quickly but the Synchronize() took a very long time.
My comment on Monday, November 01, 2010 4:36 PM, answer to question 4) has the details of what happened when I used the explicit detect changes call.
-
Thursday, November 04, 2010 5:56 PMModerator
Hi,
I can repro your scenario now. And it is a bug in FileSyncProvider metadata handling during change detection. It only happens when change detection fails in the middle. We will investigate this issue and consider a fix in the next SyncFx release. Thanks a lot for help us find this defect.
With current FileSyncProvider, is it possible for you to avoid cancelling or failing a sync during provider ChangeDetection call? If not, you may want to call FileSyncProvider.ChangeDetect explicitly, and keep a copy of the .metadata file aside to replace the bad one when change detection fails in the middle. Considering there is no data lost but just performance impact, you can decide if this workaround is needed for you or not.
Thanks,
Dong
This posting is provided AS IS with no warranties, and confers no rights.- Proposed As Answer by Dong CaoMicrosoft Employee, Moderator Thursday, November 04, 2010 5:56 PM
- Marked As Answer by jim.software Thursday, November 04, 2010 6:20 PM
-
Thursday, November 04, 2010 6:19 PM
I am currently using a work-a-round similar to the one you suggested.
1) I save the metadata after a successful synchronize().
2) If any exception happens during synchronize()
a. Delete the metadata and replace with the last know good metadata.
This has been working quite well and I have not yet experienced any performance problems.
Thank you for taking the time to reproduce the problem.
Will you let me know if or when this issue gets resolved in future releases?
Thanks, Jim
- Marked As Answer by jim.software Thursday, November 04, 2010 6:20 PM
