locked
How to use synchronization frameworks for files in a larga storage RRS feed

  • Question

  • I want to synchronize a file system storage from a local office to a server destination.

    · The destination file storage is huge. About 1 TB in 24 million files and folders.

    · The source has only a few MB.

    · The problem I am facing is that the knowlegde file on the destination machine is huge (about 200MB). I mean the syncfile.metadata

    As I understood from the sych algorithm this file is sent to the source when the source want to send files to the destination

    I am seeing that the entire 200MB is sent on the network each time it needs to tell the source it’s knowledge.

    This solution is not good for me since there are about 50 sources trying to sync the server.

     

     And if I am running the Synctoy 2.1 which is based on the synchronization framework it doesn’t send this knowledge file and is much much faster.

     

    What should I change in order to make it work normal and fast for a huge destination?

    If the  Synctoy 2.1 is capable of doing it and it is wrapped aroung the sych framework what do i have to change to use this solution in my large enviorment?

    (BTW – I am not using hash for file comparison)

     

    Here are the improtant code snippets that I am using:

     

     

    public void SyncFileSystemReplicasOneWay(string sourceReplicaRootPath, string destinationReplicaRootPath, FileSyncScopeFilter filter, FileSyncOptions options)

    {

    logger.Debug(

    "SyncFileSystemReplicasOneWay - Started");

     

    FileSyncProvider sourceProvider = null;

     

    FileSyncProvider destinationProvider = null;

     

    try

    {

    sourceProvider =

    new FileSyncProvider(sourceReplicaRootPath, filter, options);

    destinationProvider =

    new FileSyncProvider(destinationReplicaRootPath, filter, options);

     

    destinationProvider.AppliedChange +=

    new EventHandler<AppliedChangeEventArgs>(OnAppliedChange);

    destinationProvider.SkippedChange +=

    new EventHandler<SkippedChangeEventArgs>(OnSkippedChange);

    sourceProvider.DetectingChanges +=

    new EventHandler<DetectingChangesEventArgs>(sourceProvider_DetectingChanges);

    sourceProvider.DetectedChanges +=

    new EventHandler<DetectedChangesEventArgs>(sourceProvider_DetectedChanges);

    destinationProvider.DetectingChanges +=

    new EventHandler<DetectingChangesEventArgs>(destinationProvider_DetectingChanges);

    destinationProvider.DetectedChanges +=

    new EventHandler<DetectedChangesEventArgs>(destinationProvider_DetectedChanges);

     

     

    // Use SyncCallbacks for conflicting items.

     

    SyncCallbacks destinationCallbacks = destinationProvider.DestinationCallbacks;

    destinationCallbacks.ItemConflicting +=

    new EventHandler<ItemConflictingEventArgs>(OnItemConflicting);

    destinationCallbacks.ItemConstraint +=

    new EventHandler<ItemConstraintEventArgs>(OnItemConstraint);

     

     

    SyncOrchestrator agent = new SyncOrchestrator();

    agent.LocalProvider = sourceProvider;

    agent.RemoteProvider = destinationProvider;

     

    // Sync source to destination

    agent.Direction =

    SyncDirectionOrder.Upload;

    logger.Info(

    "Synchronizing changes to replica: " + destinationProvider.RootDirectoryPath);

     

    Console.WriteLine("Synchronizing changes to replica: " + destinationProvider.RootDirectoryPath);

    agent.SessionProgress +=

    new EventHandler<SyncStagedProgressEventArgs>(agent_SessionProgress);

    agent.Synchronize();

     

     

    Console.WriteLine("Synchronizing ended");

    logger.Info(

    "Synchronizing ended");

    }

     

    finally

    {

     

    // Release resources

     

    if (sourceProvider != null) sourceProvider.Dispose();

     

    if (destinationProvider != null) destinationProvider.Dispose();

    logger.Debug(

    "SyncFileSystemReplicasOneWay - Ended");

    }

    }

     

    The options are:

    fileSystemSynchronizer.Options =

    FileSyncOptions.RecycleDeletedFiles |

     

    FileSyncOptions.RecyclePreviousFileOnUpdates |

     

    FileSyncOptions.RecycleConflictLoserFiles;

     

     Thanks, Pini

     

    Monday, September 20, 2010 6:39 AM