locked
Syncing 2 remote PCs with external drive RRS feed

  • Question

  • I have 2 PCs that both need to contain a large set of files that I have (~6TB).  These PCs are in remote locations from each other but can connect via a VPN connection, although a rather slow connection.  I would like to sync these file sets but to do it over the existing network connection would be prohibitively slow as I believe there is currently about 300GB of files that need to be sync'd.  I would like to compare the file sets over the network and then have any files that need to be transferred copied to another drive (either external USB or even an internal drive that I can ship to the 2nd location). 

    Is there a way to do this with SyncToy? 

    From what I've been able to come up with so far, I'd have to sync everything on 1 PC to a large 6TB NAS and then sync that with the 2nd PC.  I don't want to do that given the volumes involved and because I don't want to have to put together a NAS just for this opperation.  I would like to be able to copy to a drive just those files that need sync'd. 

    Any suggestions would be appreciated.  Thanks.

    Tuesday, May 18, 2010 6:46 PM

All replies

  • For sure, syncToy support you have one instance on a PC, then create multiple folder pairs, left folder could be on this PC, and right folder could located on different other ones.

     

    Thanks,

    Ping


    This posting is provided "AS IS" with no warranties, and confers no rights.
    Friday, May 21, 2010 8:54 AM
  • Not sure why Microsoft would have such an answer to the above question, but I have a similar issue/question, and I don't, in any concievable way, find an answer to fishgold's question in Ping Lu's answer. Ping Lu, any comment? Or anybody else from the SyncToy gurus @ Microsoft?

    Thanks in advance!

    Nikko

    Tuesday, May 25, 2010 5:03 PM
  • // <copyright file="SyncMain.cs" company="Microsoft">
    //   Copyright (c) Microsoft Corporation. All rights reserved.
    // </copyright>
    // <summary>Sample code for using the MSF File Sync Provider</summary>
    
    using System;
    using System.IO;
    using Microsoft.Synchronization;
    using Microsoft.Synchronization.Files;
    
    public class FileSyncProviderSample
    {
      public static void Main(string[] args)
      {
        if (args.Length < 2 || 
          string.IsNullOrEmpty(args[0]) || string.IsNullOrEmpty(args[1]) ||
          !Directory.Exists(args[0]) || !Directory.Exists(args[1]))
        {
          Console.WriteLine(
           "Usage: FileSyncSample [valid directory path 1] [valid directory path 2]");
          return;
        }
    
        string replica1RootPath = args[0];
        string replica2RootPath = args[1];
    
        try
        {
          // Set options for the sync operation
          FileSyncOptions options = FileSyncOptions.ExplicitDetectChanges |
               FileSyncOptions.CompareFileStreams.RecycleDeletedFiles | FileSyncOptions.RecyclePreviousFileOnUpdates | FileSyncOptions.RecycleConflictLoserFiles;
    
          FileSyncScopeFilter filter = new FileSyncScopeFilter();
          filter.FileNameExcludes.Add("*.lnk"); // Exclude all *.lnk files
    
          // Explicitly detect changes on both replicas upfront, to avoid two change 
          // detection passes for the two-way sync
          DetectChangesOnFileSystemReplica(
            replica1RootPath, filter, options);
          DetectChangesOnFileSystemReplica(
            replica2RootPath, filter, options);
    
          // Sync in both directions
          SyncFileSystemReplicasOneWay(replica1RootPath, replica2RootPath, null, options);
          SyncFileSystemReplicasOneWay(replica2RootPath, replica1RootPath, null, options);
        }
        catch (Exception e)
        {
          Console.WriteLine("\nException from File Sync Provider:\n" + e.ToString());
        }
      }
    
      public static void DetectChangesOnFileSystemReplica(
          string replicaRootPath,
          FileSyncScopeFilter filter, FileSyncOptions options)
      {
        FileSyncProvider provider = null;
    
        try
        {
          provider = new FileSyncProvider(replicaRootPath, filter, options);
          provider.DetectChanges();
        }
        finally
        {
          // Release resources
          if (provider != null)
            provider.Dispose(); 
        }
      }
    
      public static void SyncFileSystemReplicasOneWay(
          string sourceReplicaRootPath, string destinationReplicaRootPath,
          FileSyncScopeFilter filter, FileSyncOptions options)
      {
        FileSyncProvider sourceProvider = null;
        FileSyncProvider destinationProvider = null;
    
        try
        {
          sourceProvider = new FileSyncProvider(
            sourceReplicaRootPath, filter, options);
          destinationProvider = new FileSyncProvider(
            destinationReplicaRootPath, filter, options);
    
          destinationProvider.AppliedChange += 
            new EventHandler<AppliedChangeEventArgs>(OnAppliedChange);
          destinationProvider.SkippedChange += 
            new EventHandler<SkippedChangeEventArgs>(OnSkippedChange);
    
          SyncOrchestrator agent = new SyncOrchestrator();
          agent.LocalProvider = sourceProvider;
          agent.RemoteProvider = destinationProvider;
          agent.Direction = SyncDirectionOrder.Upload; // Sync source to destination
    
          Console.WriteLine("Synchronizing changes to replica: " + 
            destinationProvider.RootDirectoryPath);
          agent.Synchronize();
        }
        finally
        {
          // Release resources
          if (sourceProvider != null) sourceProvider.Dispose(); 
          if (destinationProvider != null) destinationProvider.Dispose();
        }
      }
    
      public static void OnAppliedChange(object sender, AppliedChangeEventArgs args)
      {
        switch(args.ChangeType)
        {
          case ChangeType.Create: 
            Console.WriteLine("-- Applied CREATE for file " + args.NewFilePath); 
            break;
          case ChangeType.Delete:
            Console.WriteLine("-- Applied DELETE for file " + args.OldFilePath); 
            break;
          case ChangeType.Update:
            Console.WriteLine("-- Applied OVERWRITE for file " + args.OldFilePath); 
            break;
          case ChangeType.Rename:
            Console.WriteLine("-- Applied RENAME for file " + args.OldFilePath + 
                     " as " + args.NewFilePath); 
            break;
        }
      }
    
      public static void OnSkippedChange(object sender, SkippedChangeEventArgs args)
      {
        Console.WriteLine("-- Skipped applying " + args.ChangeType.ToString().ToUpper() 
           + " for " + (!string.IsNullOrEmpty(args.CurrentFilePath) ? 
                  args.CurrentFilePath : args.NewFilePath) + " due to error");
           
        if (args.Exception != null)
          Console.WriteLine("  [" + args.Exception.Message + "]");
      }
    }
    

    Hi Fishgold,

    You could use SyncToy (by forming 3 folder pairs on the shared drive : {PC1, SharedFolder1} {SharedFolder1, SharedFolder2}, {SharedFolder2, PC2}. Here PC1,PC2 are folders on each PC you want to Sync while shared folders are on the non-network media you physically want to transfer files with.

    However your requirement for Syncing 300GB out of 6TB of is better served by the File Sync Provider component. Where SyncToy does Not let you control Change Detection and Copying individually, File Sync Provider can.

    Check out the sample here: http://code.msdn.microsoft.com/Release/ProjectReleases.aspx?ProjectName=sync&ReleaseId=3424

    So one way I can imagine using FSP (FileSyncProvider) is:

    1) Use the FileSyncOptions.ExplicitDetectChanges option to only detect the changes, not copy. First time when you detect the changes, a special file with Metadata for the files is created underneath the folder. This is holds information like whether the file has been synced to the other side or not and more.

    (DO you expect your PCs to be in different time zones ? Normal change detection looks at file's size and timestamp to determine whether it has changed since last time. You can also use the option FileSyncOptions.CompareFileStreams if your time is note synchronized between the two PCs. However, this option scans each file and hence could be slower.)

     

    2) After performing the change detection on both ends, copy the special change detection files in two separate folders (as in PC1's change detection file goes to SharedFolder1 and so on).

    3) Now use the FileSyncProvider or Even SyncToy to do this:

    Sync PC1 with SharedFolder2 and PC2 folder with SharedFolder1. Once this is done, copy the metadata file from PC1 into SharedFolder1 and PC2 to SharedFolder2.

     

    Do you see the outline here ? Essentially, we are doing the change detection (without copying) and then transfering and comparing the generated medata file to the other location and only copying over the delta.

     

    I have not tried this but think it might work. You may want to test this out with a smaller sample. Let us know how it goes.

    (Have attached the code from the public sample.)

    Hope it helps,
    Sameer

     

     

     

    • Proposed as answer by Sameer[MSFT] Tuesday, May 25, 2010 8:19 PM
    Tuesday, May 25, 2010 8:16 PM