locked
FileSync performance grows exponentially. RRS feed

  • Question

  • I've modified the FileSyncProvider sample to just perform the detect phase on a local folder. When I run it on 250K files (occupying 9GB on an NTFS partition), the application takes 4 minutes to create the metadata for the folder from scratch. Subsequent detections take 2 minutes. Running the same sample on 1 million files (occupying 38GB on an NTFS partition), the application takes 1 hour 15 minutes to create the metadata from scratch. Subsequent detections take 30 minutes (no changes made to the files, just a like for like detect). The structure of the folders in both tests is 1000 files per folder, with files varying in size from 5KB to 150MB. The FileSyncOptions are: FileSyncOptions options = FileSyncOptions.ExplicitDetectChanges | FileSyncOptions.RecycleDeletedFiles | FileSyncOptions.RecyclePreviousFileOnUpdates | FileSyncOptions.RecycleConflictLoserFiles; Is there something I can do to boost the performance?
    Deja View The feeling you've seen this post before.
    Wednesday, October 7, 2009 9:24 AM

Answers

  • Hello,

    What you have observed is a known issue.  How many sub-folders do you have under the topmost root folder?  If there are a few, as I suggested last time, please use those sub-filders as the FSP root and do sync.  Of counse, you need to do multiple sync instead of one.

    Thanks.
    Leo Zhou ------ This posting is provided "AS IS" with no warranties, and confers no rights.
    Monday, October 12, 2009 8:39 PM
    Answerer

All replies

  • Please allow me to clarify the scenario from you.

    1. If there are total 250K files under the root folder, the change detection time (including creating metadata store) is 4 minutes.  Since destination replica does not have any file, so these 4 minutes should be all spent on the source replica.  On the subsequent sync, 2 minutes is used in change detection, which should be performed on both replicas.

    2. If there are total 1000K files under the root filter, the change detection time (including creating metadata store) is 75 minutes.  And same theory, this 75 minutes should mostly goes to the source replica.  For the subsequent sync, the change detection takes 30 minutes on both replicas.

    From your description, one quick alternative is to break 1000K files into 4 groups and each time have your app sync ~250K files.

    Thanks.


    Leo Zhou ------ This posting is provided "AS IS" with no warranties, and confers no rights.
    Friday, October 9, 2009 7:38 AM
    Answerer
  • Thanks for your reply, but that's not the issue I was querying. Querying 4 times the amount of data, yielded a 20 fold increase in the time to process the folders. Like the 250K scenario, the 1 million files are broken down into folders of 1000 files, and the tests have been run under identical conditions, but the percentage growth figures is a cause for concern.
    Deja View The feeling you've seen this post before.
    Friday, October 9, 2009 9:30 AM
  • Hello,

    What you have observed is a known issue.  How many sub-folders do you have under the topmost root folder?  If there are a few, as I suggested last time, please use those sub-filders as the FSP root and do sync.  Of counse, you need to do multiple sync instead of one.

    Thanks.
    Leo Zhou ------ This posting is provided "AS IS" with no warranties, and confers no rights.
    Monday, October 12, 2009 8:39 PM
    Answerer