locked
Large Folder Sync Across Network RRS feed

  • Question

  • I have two computers that I want to keep in sync using the file sync framework. There are about 500,000 files to sync and I have have tried to initialise sync between both but it is very slow. The indexing on the local computer is quick, about 250 secs, and results in a filesync.metadata file of about 300 MB. However after about 24 hours it is still building the remote folder index and has only created about 65 MB of the filesync.metadata. Is there a way to build the remote folder file index on the remote computer and then use that back on the local pc or is there another way of doing this faster? I realise the folder contains a lot of files and this is not a normal scenario, but I cannot avoid this.
    Tuesday, October 5, 2010 12:27 PM

All replies

  • Hi,

    Can you clarify a bit more:

    1) are 500,000 files the total combined number of files on both local and remote ?

    2) Is the remote computer much slower than your local computer that might indicate slower processing ?

    3) Since you mention building the index for the remote folder on the local computer, does it mean that the hierarchies on your local and remote computer are identical (or quite similar) even before you start sync ?

    4) One important thing to check is that your options provided for FileSyncProvider instances for local and remote computer. When the option for checking file contents is provided to FileSyncProvider constructor, the provider would scan the files and keep the hash of the files. This can be severly slow the first time index is being built, especially when the number of files are huge or the files are big. If this option was specified for the remote computer, that could explain the slow-mo.

    Coming to your question about building a faster index: The File Sync Provider builds the index in the first call to FileSyncProvider.DetectChanges. What you can explore is call this api explicitly to build the index the first time, while being cognizant of the options you provide to the provider (like check-file contents). After having built the index, then start your sync. And in subsequent syncs, you can chose to have providers not call DetectChanges explicitly, the file sync providers would call it implicitly as part of the sync work flow.

    Hop this helps

     

    Sameer

     

     

     

    Thursday, October 7, 2010 8:11 PM