locked
Multiple destination RRS feed

  • Question

  • Hello,

     

    is it possible to use Sync Framework to synchronize one source folder to multiple destination folders at the same time?
    Thank,
    Tom


    Tom
    Friday, March 13, 2009 3:04 PM

Answers

  • Hey Tom,

    This is not really a scenario that we designed for and tested. Before I look for, uh, creative solutions to your scenario I wanted to make sure I understand your scenario better and your idea of where you want to save time. You say that you don't have time for 50 sequential syncs, and it's an upload only scenario. Sounds good. In a typical sync session there are two main time-consuming stages:

    - change detection: detecting local changes in your source provider to see which files have been modified since the last change detection.
    - change application: depending on what changes the destination doesn't have, syncing the missing and updated files. Note that the set of files synced over may be different for every destination. Even if you intend to keep them perfect mirror copies there may be transient write errors that, while handled by the provider, will mean that the set of changes sent of every destination may diverge slightly. There's two big pieces of work here: reading the source file and writing it on the destination. Well, sending it over the network too.

    Now change detection is easy. It doesn't need to be run 50 times. Just run it once before anything else happens on the source provider (using the explicit change detection flag and a call to DetectChanges before the sessions) and you can use the same source replica in sequential syncs. You won't waste any time here.

    Change application is trickier. No matter what hypothetical creative solution we come up with, we can't read the source file off the disk just once and then send it to multiple destinations. Best case with what we have right now you still have to read the file 50 times. Disk cache will help no doubt. But I think there's still going to be a lot of disk thrashing which can negate any intended gains and may even make it worse than sequential. Since we're disk IO bound to a large degree, I'm not sure throwing parallel threads into the mix will help a lot here. But doing some experiments/prototypes with your intended setup will help clarify that.

    I encourage you to try the sequential sync with pre-detected changes. Also if you had a specific area from which you were expecting time savings from simultaneous syncs I'll be happy to discuss that further to see if it's doable with our current version of file sync provider.

    Lukasz
    Friday, March 20, 2009 1:39 AM

All replies

  • Hi Tom,

    I assume you're talking about MSF file sync provider. A single MSF sync session is between two providers, the source and destination, and the file sync provider cannot participate in more than one session at the same time. You can sync the same source to different destinations sequentially. If you give us more background on what you're trying to do and why you're interested in simultaneous sync to different destination we might be able to give you more info/pointers.

    Lukasz
    Wednesday, March 18, 2009 8:24 PM
  • Hi Lukasz,

     

    Thank you for the reply. Yes I'm talking about MSF File sync provider.

    I would like to develop a file synchronization tool, which is capable of syncing one source folder to multiple destinations at the same time. We have to synchronize the folders simultaneously, because there are ~50 destinations, and no time for sync the source to different destinations sequentially. It is an upload only scenario, meaning I only want the information from the source folder and I don't want to download information from the destination folders. DFS-R cannot be used because of other reasons. 

     

    We tried to do this with Microsoft Sync Framework, but according to the documentation and as you said only sync (folder) pairs are allowed.

     

    I made a test application with two SyncOrchestrator, two separate meta data file, two separate sync ID and I realized, when I call Synchronize on the first SyncOrchestrator, it made a read&write lock to the file, and the second sycnOrchestrator cannot open it.

     

    Is it possible, that during the file synchronization the SyncOrchestartor only made a write lock to the file, so the second syncOrchestrator can open it? Or Is there any workaround, to set up a hub and spoke sync topology for file synhronization?


    Tom
    Wednesday, March 18, 2009 11:00 PM
  • Hey Tom,

    This is not really a scenario that we designed for and tested. Before I look for, uh, creative solutions to your scenario I wanted to make sure I understand your scenario better and your idea of where you want to save time. You say that you don't have time for 50 sequential syncs, and it's an upload only scenario. Sounds good. In a typical sync session there are two main time-consuming stages:

    - change detection: detecting local changes in your source provider to see which files have been modified since the last change detection.
    - change application: depending on what changes the destination doesn't have, syncing the missing and updated files. Note that the set of files synced over may be different for every destination. Even if you intend to keep them perfect mirror copies there may be transient write errors that, while handled by the provider, will mean that the set of changes sent of every destination may diverge slightly. There's two big pieces of work here: reading the source file and writing it on the destination. Well, sending it over the network too.

    Now change detection is easy. It doesn't need to be run 50 times. Just run it once before anything else happens on the source provider (using the explicit change detection flag and a call to DetectChanges before the sessions) and you can use the same source replica in sequential syncs. You won't waste any time here.

    Change application is trickier. No matter what hypothetical creative solution we come up with, we can't read the source file off the disk just once and then send it to multiple destinations. Best case with what we have right now you still have to read the file 50 times. Disk cache will help no doubt. But I think there's still going to be a lot of disk thrashing which can negate any intended gains and may even make it worse than sequential. Since we're disk IO bound to a large degree, I'm not sure throwing parallel threads into the mix will help a lot here. But doing some experiments/prototypes with your intended setup will help clarify that.

    I encourage you to try the sequential sync with pre-detected changes. Also if you had a specific area from which you were expecting time savings from simultaneous syncs I'll be happy to discuss that further to see if it's doable with our current version of file sync provider.

    Lukasz
    Friday, March 20, 2009 1:39 AM