locked
FileSyncProvider Performance RRS feed

  • Question

  • I'm using the FileSyncProvider to replicate files from a network share to a local drive.  When the sync runs, it consistently hogs 95+% of the cpu and on larger shares (100GB), the folder structure gets copied over but not any of the files (as if its hanging).

     

    Are there known performance issues with the CTP 2 code?  Is there a limit on the number of directories or files the sync can handle?

     

    I'm using a sync direction of Upload.

    • Moved by Max Wang_1983 Thursday, April 21, 2011 10:03 PM forum consolidation (From:SyncFx - Technical Discussion [ReadOnly])
    Thursday, June 26, 2008 4:46 PM

Answers

  • The FileSyncProvider does not currently support syncing with an ftp endpoint. If you want to sync a local folder with an ftp folder, today you will have to write your own provider that does that.

     

    Tuesday, July 8, 2008 6:46 PM
    Moderator

All replies

  • The FileSyncProvider will take more time to sync if there are a lot of files in the sync directories.

    How many files do you have in the larger share? Do you see sync successfully copy the files even after long time?

    Thursday, June 26, 2008 11:08 PM
    Answerer
  • The large 100GB share has 58,277 files in 514 folders.  The metadata file is 39,444KB.

     

    We started to see files appear in the target location 4 hours after the sync started.

     

    Monday, June 30, 2008 6:15 PM
  •  

    Thank you for the information.

     

    I assume you didn't turn on hash option for the file sync.

     

    The share does slow down the file transferring process a lot.

    I tried to copy one 1GB file between directories on same machine and it took 80 seconds.

    then I tried to copy same 1GB file over to a network share and it took 170 seconds.

    Sync one 1GB file with FileSyncProvider for directories on one test machine took about 90 seconds.

    Sync 100,000 small size files in certain number of directories took about 16 minutes.

     

    More analysis is needed to see why the first file from a share appear after so long time.

    Tuesday, July 1, 2008 12:29 AM
    Answerer
  • Some more info:

    • The hash option is off for the file sync. 
    • The source network share is in a different doman than the target location so I'm using impersonation while doing the sync.
    • I'm storing the metadata files for the Source and Target locations on the local machine that's executing the sync.

    Let me know if you need any more information.

     

    Tuesday, July 1, 2008 7:13 PM
  •  

    Could you try (x)copy all the dirs/files from the remote share to local machine and see how much time that takes?

    Sync will definitely take more time for supporting different logics but if the copy takes more than 1-2 hours, then the sync delay could be to some extent expected.

    Wednesday, July 2, 2008 7:20 PM
    Answerer
  • It seems like it may be a network speed issue between the local machine and the network share.  Doing a plain xcopy we're able to copy about 2GB per hour.

     

    Monday, July 7, 2008 2:31 PM
  • I have a question about to sync with a remote loctaion. Is possible to sync a local folder with the contents of an ftp folder?

    Is it possible with a FileSyncProvider and there are some samples in any place?

     

    Tuesday, July 8, 2008 9:16 AM
  • The FileSyncProvider does not currently support syncing with an ftp endpoint. If you want to sync a local folder with an ftp folder, today you will have to write your own provider that does that.

     

    Tuesday, July 8, 2008 6:46 PM
    Moderator