locked
Synctoy 2.1 - Fails testing : Performs vast number of needless file copies when renaming a subfolder under certain circumstances RRS feed

  • Question

  • Synctoy 1.4 used to crash and Synctoy 2.0 got stuck at about 60-70% when attempting to sync a 'small' 30 gig folder.

    OK, it's Nov09 and SYNCTOY 2.1 is released.  !!!!

    Let's test to see if it can handle copying data from my source folder to a destination. 

    The main reason I would like to use Synctoy rather than using Robocopy is to avoid unnecessarily recopying files and folders that you may have moved or renamed -  which should provide a vast speed improvement over using a simpler utility like robocopy.
    To summarise the results below for Synctoy 2.1 testing:

    -> Good news is that Synctoy no longer crashes, and no longer gets stuck.

    -> Bad news is that if you use Synctoy 2.1 and create a new folder pair for a source and destination folder that ALREADY EXIST  (and are already perfectly synchronised using something like robocopy or perhaps by using synctoy 1,2 or 2.1 using a previous folder pair),  then it completely fails to recognise subfolders that you subsequently move or rename and it ends up copying them all over again,  leaving BOTH in the destination.

    ------------------------------------------------------------------------------------------
    The following tests are using the 'Echo' feature of Synctoy 2.1.
    This PC is an Intel Quad core 2.5GHz with 3g ram running Vista Ultimate 32bit.

    My test folder is 184g (242,616 files in 9096 folders)
    I'm still using robocopy to handle my backup to external usb drive; once the drives are synced  if I do a second pass robocopy takes just 4.5 seconds to whizz through the 242,616 files and show that there's nothing left to copy.

    Using Synctoy 2.1 I created a new folder pair for this pair of folders using echo with standard options, and clicked Preview.

    On the first pass I did think it was stuck at 60% but it carried on without problem taking 1 hour and 22 minutes. That's quite reasonable considering synctoy 2.0 got completely stuck on a folder a sixth of the size.  It has now created a database representing the file structure which should allow it to identify folders or files that were renamed, ultimately avoiding hours and hours of unnecessary copying and network traffic. It looks promising.

    Strangely, out of the 242,616 files it did identify 15 files from two folders that needed updating,  even though I ensured the folders were completely in sync before I started.

    I just used WINDIFF to check the source and target subfolders that contain these particular 15 files. Yup - the files in this folder are definitely identical, so I don't know why synctoy has identified that these files need to be overwritten. (The source and destination drives are both local and both NTFS).

    OK, I let it run anyway,  despite showing me it had 15 files to copy from d: to f:,  it just actually only did 6 overwrite operations -  no other operations, (and 485,146 files did not require action which is presumably around 242+ thousand files on D and a similar number on F)

    I windiffed the two folders again and all the files still match.

    OK, let's see how long synctoy 2.1 takes now  on the second pass... there should be nothing to sync.  Wow... it took 2 mins 40 seconds - that's not bad at all !   Did another preview - now 2 mins 5 seconds.

    So it's thumbs up for Synctoy 2.1 in terms of speed of working out its tasks, not getting stuck and not crashing.

    Right, here's the big test for Synctoy 2.1:
    Inside this 184g folder is a subfolder that contains 11.7gb and over 56,500 files including 1799 folders.   I'll now rename that subfolder on the source drive and then attempt to sync again with Synctoy.    Other methods of syncing including the robocopy script with the /MIR switch  would delete the 11.7 gb from the destination and copy the 11.7gb all over again.   Synctoy should recognise the name change and just rename the destination folder taking no time at all, though I appreciate it might accomplish it by doing 56,500 separate move operations - (which should still be a lot quicker than copying the 56,500 files! )


    I clicked Preview to kick of the echo operation again. It came up with a vast number of delete and new operations and only a few renames. The preview took 35 mins 29 seconds and in total it found 76,738 actions to perform !!

    That's a big fail for Synctoy 2.1.  What's the point of spending well over an hour to make that database on the first pass if it's not going to detect a single folder rename?

    I then let it run - but I eventually stopped it when it had done over 19,000 operations after about an hour. I manually moved the folders back that it was copying unnecessarily and then deleted the 13186 files (2gig) that it had created so far in the destinations recycle bin.
    OK, I can't trust synctoy now to get this right because I've messed about manually with the destination folder.  To get the destination perfectly identical to the source without deleting the destination and starting all over again, I used Robocopy with the /mir switch.  OK, the destination is now exactly the same as the source.

    I then ran the preview once again on that large area but it now says there are 323 folders to delete, 6186 files to delete, 6186 new files and 323 new folders.   OK, actually, while doing my previous testing, I did do a few renames of other subfolders in both the source and the destination.  They do match perfectly, confirmed by robocopy, but of course synctoy has a database that reflects what it saw in the source folder the last time I ran it... so presumably it sees the changes I made and thinks it has a load of work to do, when in fact there is none to do. I'm not sure I like this idea of how synctoy checks against a database that it created last time - I think it really needs to check all of the destination files and folders each time.  It's really making the assumption that the destination is only ever changed by synctoy,  not allowing for manual copying or editing of files, nor for renaming of folders on anything but the original source.

    If I could remember all of the folder names that I renamed and put them all back again, I'm sure Synctoy would then be happy. But the only way to sort this mess out now and avoid Synctoy attempting to perform all unnecessary moves and renames is to delete this folder pair, recreate it and run the preview again. I did this and after waiting for the hour and a half processing, confirmed it now shows 0 files to copy.


    Let's test Synctoy 2.1 with something much much simpler.
    I created a folder d:\test with 3 subfolders a,b, and c. They all have files, but folder b has 2 further subfolders called fff with 12 files and 2 further subfolders,   and ggg in which there are 7 files.
    Let's sync that to f:\test using echo.
    The preview took under a second. The run took 14 seconds.
    Ran it again - took under a second to show that there are zero files to copy.
    Right,  I now manually rename folder b to t
    The preview again takes under a second and shows 5 folders to delete, 67 renames and 5 folders to create and it ran this in 1.5 seconds.
    Well that worked perfectly... That said, it's a shame it doesn't realise it can just perform a single rename on the top level folder and instead do individuals moves of every file within!   But what went wrong with my large folder .. is it just a matter of folder size that confuses Synctoy ?

    I'll try a smaller folder within the 140 gig structure. This folder contains 17 files, 164meg and includes one subfolder.  OK, it got that right....  just rename operations and create folders.
    Let's try another folder -  194 files, 1.25 gig including 74 subfolders -  it got that right too apart from 2 files which it decided to delete and recopy.
    Now try another folder - 902 files, 5.22 gig including 572 subfolders.  Now it's starting to go wrong.  It came up with 209 folders to delete and create, 332 files to delete, 571 files to rename (move), 332 smallish files to delete and recopy - (totalling 75 meg).  In this case that wouldn't take too long,  but why is it doing this?
    Now a bigger folder -  6760 files , 15.99 gig including 2162 subfolders.  It came up with 739 folders to delete and recreate, 5775 files to rename (move), 1051 files to delete and re-copy totalling 103meg.  Again, interesting it is choosing small files to recopy. I checked out some of these files - they're not read only.  Perhaps I have a clue.... these files are all from a small set of subfolders.

    I'll try setting up a new sync pair using one of these subfolders that it decided to delete/copy rather than rename/move. 
    This folder structure is 1174 files, 80 meg and contains 389 subfolders.  I'll start off with the folder names being the same; the preview ran very quick and showed nothing to change. OK, change the subfolder in question inside the source folder - YES - it again says these are new files. The patch is very long... I'll subst a pair of drive letters for the source and destination a long way down the path so that the folders are no longer long.  This'll tell us if it's something to do with the contents of the folders, or just a problem with path length.  It still says 179 new files to copy.
    I substed a pair of drives letters as close as possible to the folder in question and it still showed 179 new files to copy.
    In case it's something to do with the files themselves, or their names, I'll copy them to a simple test folder d:\test - then I'll rename the single subfolder inside.  It still shows new files to copy.  This is now a simple folder with a single subfolder. Still the same problem.

    Now let's just go back to creating a very small test folder with just 2 subfolders and 1 file  similar to the successful test I did before.  Well now this causing trouble with synctoy.  I can't work out why sometimes it successfully renames/moves files, and other times it wants to copy a new file  (and sometimes doesn't delete the original leaving 2 copies in the destination).  [EDIT - I've since discovered this occurs only if the destination folder already exists when you created the synctoy pair. It never goes wrong if you let synctoy copy the entire source to an empty destination folder on the first run.....   but this means you can never delete and recreate the folder pair unless you delete the entire destination folder as well].

    I'll try one more - with my pics folder - 28024 files, 62 gig,    already robocopied to the destination folder. The preview operation successfully shows nothing to copy. Renamed a subfolder one level down, and another subfolder at level 2. It's got the whole lot wrong, synctoy wants to copy 1608 files totalling 4.6 gig. If I completely close synctoy, reload and try again, the results are the same.

    I'm afraid Synctoy 2.1 echo just doesn't work !  I really was hoping that Synctoy2.1 would be a winner. The fault may lie within Synctoy itself or within Microsoft Sync Framework on which Synctoy is based - so you'd have to test other products based on sync framework to see if the same problem exists in them.
    Most people will assume that synctoy would work without problem so I guess a lot of time is going to be wasted; I've just spent 6 hours trying to get it to work properly myself. I hope MS can come up with an update to Synctoy and/or Sync Framework asap.

    Tim from howeasyisthat.com and brisbanepc.com.au

    [I've made a number of edits mainly to make this easier to read,
    and to include a summary of the findings right at the top]
    Friday, November 20, 2009 3:35 AM

Answers

  • Hi Tim -

    Thanks for taking the time to test SyncToy 2.1. We use a heuristic to determine a rename as opposed to a delete/new of a file. The heuristic unfortunately is exactly that - a heuristic and if it fails to beyond doubt identify a rename, we then tag it as a delete/new of a file because mistakenly identifying a rename would result in data being lost. The current behaviour while not best with respect to performance will not result in data being lost. The only reason that you see this behaviour on a large dataset is because it just makes it more likely that we are unable to accurately identify a rename.

    We appreciate the time and effort you have taken to identify this problem. For a future release we will attempt to see if we can identify a rename more accurately.

    Thanks
    Deepa

    Deepa ( Microsoft Sync Framework)
    Wednesday, December 30, 2009 10:30 PM

All replies

  • I have hit exactly the same scenario - synced my folder pairs with Synctoy 2.0 as instructed.  Installed 2.1.  realised that one of my folders contained about 90 files I did not want so I deleted them.  Reran Synctoy 2.1 - it did not notice the deletion. 

    To confirm the setup - the left folder was the one I deleted the files from, the mode was Echo and the right folder still contains the files.

    The left folder is a local internal disk, the right folder is a local external USB connected disk. 

    Not tried it with my NAS - which is why I upgraded from 2.0 in the first place.

    I can also confirm the apparent hanging - but noticed a lot of sisk activity for some of the time so it might just not be reporting what it is doing - still not a good thing in this sort of utility.
    Friday, November 20, 2009 11:17 PM
  • ok,
    Let's be fair and try the test again. This time instead of a Vista SP2 x86 pc,  I'll use a Win7 x64 pc  (both similar specs, quad core 2.5ghz).  The win7 pc has never had synctoy installed.

    The external usb drive is v: and this is now my source (v:\test) with  242,622 files (184gb) including 9096 subfolders.
    The local drive is d: and this is my destination. (d:\test)

    (First I'll just delete the existing hidden v:\test\synctoy_xxxx.dat file left from my previous testing with the vista pc.)

    First I'll copy v:\test to d:\test using robocopy.
    Now I'll run robocopy again using robocopy v:\test d:\test /mir  and confirm there are no files left to copy.

    All good, now ready to instal and test Synctoy 2.1 x64.
    I created a new folder pair  v:\test   d:\test,   configured to ECHO with all standard options.

    The sync operation took 21 mins 20 secs to preview.  Excellent. 
    It found 8 files to copy which robocopy hadn't.  The target files were named with capitals whereas the source files were mixed case. Interesting that robocopy did this. Strange synctoy doesn't want to rename them but copy again.  Never mind, this doesn't count against synctoy.

    After allowing this to run, I ran the synctoy preview again - great - it took 1 min 52 seconds and found 0 files to copy.

    Now ready to test. I'll rename a subfolder within v:\test .  This folder contains 11.7gb consisting of 56,563 files and 600 subfolders.

    I ran the preview on v:\test.
    It took 15 minutes 12 seconds to come up with the preview results :

    Delete folder :  601
    Delete: 18,973
    overwrite : 0
    Rename: 37,590
    New: 18,973
    Create folder 601
    Total actions to perform : 76738
    Total bytes for copy : 2,084,282,166  (2g) 


    So what are these 18.973 files (2g) that it wants to recopy?     I'm thinking this might be something to do with timestamps and sub-2 second granularity, yet both file systems are NTFS. There's something that SYNCTOY is seeing to make it think these files are different, which does not fool robocopy.  Just to add, when I use ROBOCOPY I am NOT using the /FFT switch to overcome such problems.
    The paths lengths aren't excessive (e.g. 57 characters) and just 6 levels deep underneath the test folder; the files are a mixture of types, nothing significant, the timestamps are a mixture . 


    I picked on a subfolder in which synctoy wants to recopy quite a few files and windiffed these two folders. Windiff shows all identical. I picked on an individual file and from the command prompt using FC against the D: and F: drive: 'no differences encountered'.  Also used windiff and showed these files are identical.
    Using explorer and viewing properties. Created, modified and accessed attributes are all the same.
    CACLS to check permissions on each file : indeed permissions are different but identical to all the other file permissions throughout the test structure - we've got full access and remember, robocopy was quite happy to see these files were identical. Permissions are not the issue.
    Systernals SIGCHECK of course also shows the same file version, author, details etc.

    I then let Synctoy run the sync.
    Copied 2,084,282,166 bytes in 18,973 files in 01:18:18:29.
    Avoided copying 381,373,706,798 bytes in 409,632 files that did not require action.
    Saved approximately 22:47:06:895 by not copying all files.

    Hmmm, so it asserts that it saved 22 hours 47 mins by avoiding copying 381gig.  That statement is of course incorrect - it's counting the size of the source and destination to come up with 381gig - it really means it's avoided copying half that.  But the main problem is, it spent 1 hour 22 minutes copying 2 gig of files which was completely unnecessary.

    I ran the preview again - now correct shows 0 files to copy in 6 minutes, 1 second

    These are the very same results as when I tested on Vista x86 (though much quicker - 15m 12s versus 1h225m.)  Just re-reading the text I submitted earlier for Vista with Synctoy x86, I notice I didn't quote the actual preview results; the preview results for the test under Vista x86 were the same - EXACTLY THE SAME with 76738 total actions.  I'm betting therefore it was exactly the same set of files and folders that it decided it needed to re-copy rather than rename/move, though I don't have a record of the actual files/folders that it decided to recopy. (The log, even in verbose mode doesn't show what files and folders it will copy).


    SO, if
        1) SYNCTOY can't be relied upon to figure out it doesn't have to copy files/folders - this means when attempting to echo a large folder structure over a network, it will not finish in any sensible time because it will delete and recopy, despite the fact that this is one of its primary features and really the only reason I would want to use synctoy rather than robocopy

        2) Even if it did work, when using the ECHO faciliy, you must never ever touch the destination folder, never delete, edit or copy anything else in there. But if there is the slight possiblity that the destination folder may be ever be played about with  (isn't there always ?) then SYNCTOY is not suitable.  The danger is that should a file or folder ever go missing from the destination (slip of the mouse, working on the wrong drive, other user or administrator tinkering),  then using SYNCTOY will NEVER KNOW about it!  When synctoy keeps your files synchronised from the source to the destination, SYNCTOY doesn't keep tabs on the destination folder; once a file goes missing from it, synctoy won't look again at the destination folder and will therefore never know that the file is missing (Remember I'm talking about use of the ECHO facility here, not Synchronise).... not unless you coincidentally update the file in the source in which case it will then indeed get copied over to the destination. To sort this mess out, you really have to delete the folder pair and recreate it, thus waiting for a long time while it builds up its database again. 

    I want to use synctoy, I really do.  However I don't want to recopy ISOs and large folder structures unnecssarily just because I renamed something, but it's far more important to me to know my destination (backup drive) is a good backup. Synctoy 2.1 fails on the above two counts.
       
    Perhaps if I had let synctoy create the destination folder from scratch rather than using robocopy,  then I wouldn't find any problem.
    Does this mean robocopy is to blame?  I don't think so.  I proved above that robocopy worked correctly becasue the example files/folders are indeed identical as showned by FC and windiff, as well as viewing the attributes of the files.
       
    It seems that when SYNCTOY gets a folder pair for the first time it uses one method to match up files,  but perhaps when you are doing a rename/move it uses different code and checks in a different way.  Could that explain why synctoy initially is happy that there are 0 files to copy,  yet when renaming a subfolder it subsequently finds a large number of files to recopy rather than rename.  What if I repeat the excercise, but this time use the same folder structure to test.  SYNCTOY has already copied about 2000 files last time, so hopefully it copied them in a way that synctoy is happy with, though I couldn't identify the difference.  I'll rename the folder again and see what synctoy has to think about that.  Surely this time it will just do rename/move operations correctly.?

    OK, here's the test... created new folder pair for the d:\test and v:\test folders, 242,622 files (184gb) including 9096 subfolders as before.
    Run the SYNC preview using ECHO - confirmed 0 files to copy (taking 20 mins 53 secs)
    Rename that subfolder contains 11.7gb consisting of 56,563 files and 600 subfolders (of which it previously unnecessarily copied about 2000 files totalling 2gig).
    Recreated the folder pair and run the folder pair echo operation.
    Good grief... now it says 56,563 new files to copy and 601 folders to create (no delete operations, overwrite or renames). It seems to think it's doing a new folder that doesn't previously exist. In which case, how come it shows 0 files to copy beforehand? Let's check I got this right.  Left folder : v:\test  Right folder e:\test.. That's correct v:\test is my source and that's the folder in which I have just renamed a subfolder.
    One more time just in case I did something stupid, I'm sure I didnt.
    I'll get both subfolders named back to their original, delete the folder pair,  delete the hidden synctoy*.dat files.
    OK,  ROBOCOPY v:\test e:\test /mir /ndl   ... good 0 files to copy, 0 extra files. Just to be paranoid I'll do it in the other direction - good.. that's 0 files to copy. 9095 dirs, 242613 files, 184.438g.

    Now delete the folder pair again.  Create the folder pair   v:\test e:\test ,   choose ECHO
    Run Preview.  It took 19 mins 48 secs and correctly shows 0 files to copy. The usual message that doubles the number of files and bytes in question appears because it is taking into account both source and destination : ie : Found 485,160 files that did not require action. Avoided copying 396,005,600,480 bytes in 485,160 files
    I repeated running the Preview.... again 0 files to copy... took 12 mins 28 secs.
    Now I rename the 11.7g subfolder (56,563 files and 600 subfolders )... this time I'll turn off AV just in case.
    Running the Preview. It took 22 mins 35 seconds to show preview results.  Same again, 56,563 new files to copy and 601 folders to create.


    Synctoy 2.1 just doesn't work when you have a large folder structure. How large ?  No idea, but I want a utility that works reliably and doesn't give strange results. Just have to stick with Robocopy for the moment!

    Tim from howeasyisthat.com and brisbanepc.com.au
    Saturday, November 21, 2009 10:02 AM

  • It looks like Tim from howeasyisthat.com has done some decent testing here.

    I agree, Synctoy's ability to recognize when you've moved or renamed things so it doesn't have to re-copy multiple gigabytes is very desirable feature.
    If Synctoy can't do that, then why don't I just carry on with xcopy or robocopy, (or sometimes xxcopy)

    The other advantage is that synctoy is GUI based, so maybe that's a good feature for non techies who don't want to learn the switches.

    But, and this is a huge but, if it still has known errors after all this time then is it trustworthy?
    Also if you mess with the destination, then it NEVER get corrected because synctoy never re-echos the files. In the real world, robocopy is a better option because it would recopy the files over.

    If Synctoy 1.4 crashed,  Synctoy 2.0 got stuck and Synctoy 2.1 is inconsistent then I wonder why are we still all looking at it considering robocopy does work with any size folder and really has always worked (vista version, xp resource kit version, win7 version).


    That 180 gig folder that was tested is pretty big.
    I decided to test myself with a 55 gig folder  again using ECHO

    source networked XPSP3 pc  (workgroup).  c:\data  25,000 docs and images ranging from 1m to 40meg
    target networked XPSP3 pc (workgroup).   y:\data 

    Synctoy testing using ECHO
    Rename source folders:
    \data\2008\sep_oct to \data\2008\September_October.   1030 meg,203 files.
    \data\2008\nydata to \data\2008\ny    3320 meg (1048 files)
    \data\2006   to \data\Y2006 .   6726 meg (2208 files)

    The preview confirms the previous findings
    It didn't find one rename or move operation. Instead it found 3456 new files.
    That is completely useless! It would be quicker and safer to use robocopy.
    Synctoy doesn't work!  Shouldn't we all be asking MS to go back and come back when it's fixed before we can trust it?  What's the point of all these questions in the forum if the utility actually doesn't work?

    What do you think?  Should MS come out with v2.2 or is that going to take another year?
    Have you done any testing ?

    How about adding your concise test results below (whether successful or not.)?  e.g.
    Remember, if you've messed about with the destination and you are using echo, you need to delete the folder pair and recreate it, or get them back in sync by some other method.

    SYNC, ECHO or CONTRIBUTE
    Folder size :
    Number of folders :
    Number of files :
    What has changed before running folder pair
    Operation expected :
    Results :


    Maybe we can find what size folders, or what scenario cause SYNCTOY to fail.

    Tuesday, November 24, 2009 11:03 AM
  • Can someone from the synctoy team say how large a folder has been tested? Do you acknowledge the problem?
    Many thanks

    Wednesday, November 25, 2009 4:00 AM
  • Anyone else done any testing, anyone ? MS, how about yourselves?! Thanks, Tim
    Friday, November 27, 2009 11:16 PM
  • I haven't been able to confirm this but it looks to me from your above testing that it has to do with file sizes whether it rename/moves files or deletes and copies new over. Does this sound possibly correct? Although I don't understand the logic involved in doing this.
    Monday, November 30, 2009 8:44 AM
  • Using SyncToy v2.1, I was trying to sync a folder containing TomTom files to a Network Drive on my Home Wireles LAN. Some files are as large as 2 GB and the total folder size is around 3.5 GB. These are worth backing up as I have paid for maps and updates to the maps and do not want to lose them if my LapTop HDD crashes. I got various exception errors during the sync. I am using action "Contribute", so it should be a one-way copy to the Network Drive, about half are new and half are replacements, as I manually copied the TomTom folder to the Nework Drive about 6 months ago. 

    After several attempts with the same result (lots of exception errors), I deleted the Folder Pair and started again. This time it worked fine. I was sure it would fail again, as I am using a Network Drive (FreeCom), and I have seen lots of issues with these. So my advice is to give it a another try after deleting the Folder Pair. I hope that it works reliabally in the future. I will report back if it fails again, but for now all is well.
    Monday, November 30, 2009 4:00 PM
  • Hi Jostrus

    I tried to look for a pattern that might cause the unnecessary re-copying of files by Synctoy 2.1 e.g. file type, file sizes, folder sizes, number of files, number of folders etc.
    I agree, at first it seems to be related to file sizes but it appears to be variable. Yet once it starts to go wrong, then synctoy starts insisting on recopying all files and folders regardless of their size, leaving the original file/folder names in place on the destination.

    It's no longer dependant on the file size or anything else, it never recovers even if you try to delete folders and reduce the size of the source folder to a minimal size.


    After further tests (several hours) I've established that if synctoy 2.1 created the destination folder in the first place (using the same folder pair) then when you perform successive synctoy echo operations, you will never encounter the copy problem.
    If you create another folder pair (either using the same set of folders, or using a pair of folders previously synchronised,e.g. by robocopy),  then synctoy will now exhibit the problem whereby if you rename files or subfolders, it copies all of the changes to the destination while leaving the original folders/files in the destination. Presumably when Synctoy 2.1 first encounters a folder pair with an existing destination folder, there is a bug when it builds up its initial database which I believe is a one time operation. 

    IT seems I may have made an assumption here. The documentation states that if you have a previous version of synctoy you must synchronise the folders before upgrading.  I have assumed that you can initially synchronise the folders by any means. In fact it will only work reliably if the destination folder is initially EMPTY and you allow synctoy 2.1 to populate it during the first sync. That's a big problem if you already have a synched folder pair and are considering switching to use synctoy 2.1 to take over the synching role. In the real world, we have destination folders of hundreds of gigabytes that are already fully synched using other methods.... it's crazy that you can't switch to synctoy and use an existing synchronised folder pair without risking re-copying huge folders while also leaving the originals during what should have been a simple rename operation.

    Often, to solve a problem encountered with synctoy, re-creating the folder pair is the answer. In fact, this introduces the very problem discussed here, resulting in subsequent rename/move operations causing large amounts of unnecessary copying that also resulting in duplicated data in the destination.

    Here's a concise summary of the previous tests I did (1,2,3) followed by a whole new set of tests 4) :


    1)  Folder source contains several hundred files, 5 folders.
    Destination is already synchronised and Synctoy 2.1 preview using ECHO now confirms 0 files to copy
     
    Test: Rename a subfolder containing 67 files and 2 subfolders. Synctoy 2.1 preview successfully sees 67 rename operations to perform and creation of the necessary folders, and SUCCESSFLLY avoids copying any files.

    2) Folder source contains 184g, 252,000 files. 
    Folder pair is already synchronised, Synctoy preview using ECHO shows 0 files to copy.

    Test:Rename a subfolder within the source containing 17 files, 164 meg - 
    Result:Synctoy 2.1 preview successfully found all rename operations - no files to copy.

    Test:Rename a subfolder within the source containing 194 files, 1.25 gig including 74 subfolders. 
    Result:Almost perfect - Synctoy 2.1 preview decides to successfully rename most files but decides to delete and re-copy 2 files

    Test:Rename a subfolder within the source containing 902 files, 5.22 gig including 572 subfolders.
    Result:  Synctoy 2.1 preview decides to delete and unnecessarily recopy 332 files totalling 75 meg averaging 221k each.

    Test:Rename a subfolder within the source containing 11.7gb folder with 56563 files,600 folders.
    Result Synctoy 2.1 preview decides to delete and unnecessarily recopy 18973 files totalling 2g averaging 111k each.

    Test:Rename a subfolder within the source containing 6760 files , 15.99 gig including 2162 subfolders. 
    Result:Synctoy 2.1 preview decides to delete and unnecessarily recopy 1051 files totalling 103meg averaging 100k each


    3) Folder source contains 62gig, 28024 files
    Folder pair is already synchronised, Synctoy preview shows 0 files to copy.

    Test:Rename 2 subfolders within the source totalling 4.6 g in 1608 files.  
    Result:Synctoy 2.1 preview decides to delete and unnecesarily recopy all 1608 files averaging 3meg each.



    4) - This is a new set of tests using an XP SP3 x86 pc which has never had Synctoy installed.

    Created a new Synctoy 2.1 folder pair using ECHO.
    The destination drive is another folder on the same local HDD
    The source folder c:\test_souce contains a combination of 10k-500k docs, 300k to 4meg jpgs, 1.5meg to 20meg mpg files, 23k to 166meg zip

    files.
    I've purposely mixed doc,jpg,exe and zip in the same folders in order to check if synctoy treats different file extensions differently.
    I left the default options (as with all the other tests) and used ECHO.

    The Folder source contains  10.2g, 5667 files, 88 folders.
    c:\test_source
        \pics1        4.76g, 2165 files, 15 folders : All jpgs
        \pics2        4.08g, 1364 files, 25 folders: All jpgs
        \combo        8 files, 0 folders : mpg,jpg,zip,exe,doc,docx files
        \user        2138 files, 44 folders: doc,docx,xls,xlsx,jpg,gif,eml

    Used synctoy 2.1 to syncrhonise to c:\test_dest folder  (rather than using robocopy).
    Synctoy preview now shows 0 files to copy

    TEST: Rename the combo folder and rename the pics1 folder
    Result:  Synctoy 2.1 SUCCESSFULLY identified 1371 files to rename - no files copy or delete


    Another test:

    c:\source 11.9g 6234 files, 91 folders  (again a mixture of file types)
        \data1        (10.4g,5677 files,87 folders)
                root contains 2.56g, 1159 files
            \d1    4.28g, 1371 files,  26 folders
            \d2    1.78m, 13 files,  0 folders
            \d3    3.23g, 3023 files,  51 folders
            \d4    308m, 77 files, 3 folders
            \d5    75.8m, 34 files,  2 folders
        \ffff        1.53g, 557 files,2 folders
                Root contains 1.46g, 550 files
            \b1    3.56m, 3 files,   0 folders
            \b2    60.5m, 4 files,  0 folders

    Used robocopy to create the destination folder in perfect sync with the source.
    Then used synctoy 2.1 to create  a new folder pair c:\source and c:\dest using ECHO
    Ran Preview : Synctoy preview now shows 0 files to copy


    TEST: Rename D3 to Z3  and ffff to ZZZZ
    Result : Synctoy 2.1 preview failed to identify any renames at all - it decided to copy 3580 files, no deletes,copies or renames.

    Test: Reverse the above renames: rename Z3 to D3 and ZZZZ to ffff
    Result : Synctoy 2.1 preview now successfully shows 0 files to copy

    Test: rename d1 to DDD1
    Result : Synctoy 2.1 preview shows 1371 files to copy

    We seem to have reached the same problem as the last set of tests - it has decided to delete/copy files rather than rename no matter how simple the operation.


    Test: Clear recycle bin, repeat test. 
    Result: 1371 files to copy

    Test : Changed option NOT to save overwritten files in the recycle bin
    Result:1371 files to copy

    Test : renamed ddd1 back to d1 and put back the default option of saving overwritten files in the recycle bin
    Result : Synctoy 2.1 preview now successfully shows 0 files to copy

    Test:  rename b2 to BBB2
    Result: Synctoy 2.1 preview shows 4 new files to copy (+ 1 folder)

    Rename bbb2 back to b2. Confirm folders are in sync with robocopy source dest /s /l

    Test: Without changing source or destination folder, delete folder pair and recreate folder pair
    Result : Synctoy 2.1 preview shows 0 files to copy

    Test: rename b2 to bbb2
    Result: Synctoy 2.1 preview shows 4 new files to copy (+ 1 folder)

    Test: Since the above test only involves 4 files, and since it showed only copy operations, not delete, let's see what happens if we run this operation.  Will it leave the old folder behind in the destination as well as copying the new folder name ?
    Result: It did indeed copy the 4 files to bbb2 and left the original copy in b2. This was a simple folder with 4 files sized from 8meg to 31 meg.

    I hadn't highlighted this issue before, but this really means there's another serious problem with Synctoy 2.1.  Not only can it fail to recognise folder moves or renames (even if it's a small folder with no subfolders or even a single file), but it also ends up duplicating data in the destination - thus risking filling your destination area earlier than you might expect.


    TEST :    Now MOVE ffff from the source to c:\ and rename as ffff_source.  Move ffff from the destination to c:\ and rename as ffff_dest
        robocopy c:\ffff_source c:\ffff_dest /l /s and confirm fully synchronised.
        Create new synctoy 2.1 folder pair for c:\ffff_course and c:\ffff_dest.
    Result:    Synctoy 2.1 successfully shows 0 files to copy.

    Test:    Rename c:\ffff_source\b1 to bbb1
    Result : Synctoy 2.1 says 3 new files to copy, 1 folder to create.

    Test:    Reboot, try again. Run preview
    Result : Synctoy 2.1 still says 3 new files to copy, 1 folder to create.

    Test:    Delete folder pair, rename folder bbb1 back to b1, confirm synchronised with robocopy
    Result:    Synctoy 2.1 shows 0 files to copy

    Test :  rename a single file in the root of c:\ffff_source
    Result: Synctoy shows a NEW file to copy.

    It certainly seems that once synctoy starts going wrong,  it continues to do so even after a reboot, and even after deleting and creating new folder pairs and even after removing and reinstalling.

    I'll repeat a test that worked earlier:

    TEST: Using the c:\source and c:\dest folder, confirm already synced with robocopy and then with synctoy using a new folder pair.
        Rename the combo folder and rename the pics1 folder
    Result:  Synctoy 2.1 finds only new files to copy - no renames.

    Uninstalled SYNCTOY,  Sync Framework 2.0 provider services and Sync Framework 2.0 core components
    Rebooted, reinstalled synctoy 2.1

    TEST: re-ran preview to see if it detects the rename of the combo folder - but again it just finds new files.

    Deleted the hidden files synctoy*.dat from the root of the source and destination folders.
    Deleted the contents of %userprofile%\local settings\application data\microsoft\synctoy\2.0
    renamed comboxxxx folder back to combo, confirmed synchonised using robocopy and then synctoy using new folder pair

    RESULT: Re-ran preview - it still finds new files to copy, no renames.

    TEST: I created muliple further tests using new smaller folder pairs, each originally synched by synctoy.
    Result:  Preview shows successful renames

    TEST: Copied more folders in to the source including a copy of the combo folder referred to above.
    RESULT:  Synctoy 2.1 continues to show successful renames for multiple and complex rename and move operations.

    TEST: Copied more folders increase overall size of folder structure to 800meg and used echo to synchronise.
    RESULT:Synctoy 2.1 still working

    TEST:Copied more folders to increase overall size of folder structure to 1.4gig and used echo synchronise.
    RESULT:Synctoy 2.1 still working

    TEST:Copied more folder to increase overall size of folder structure to 1.9gig and used echo synchronise.
    RESULT:Synctoy 2.1 still working


    Final test - Created new 14.1gb source folder, 6888 files, 120 folders. 
    TET:Used ECHO to sync to an empty destination folder using Synctoy, then performed extensive move and rename operations
    RESULT:and synctoy flawlessly performed rename operations for each.


    IT appears to work if you actually run the operation to synchronise. So even if it was going wrong originally, once you run the operation and perform the unnecessary copies, from then on it will work successfully.

    However if you create a new folder pair and refer to an existing source and destination folder that an existing folder pair is successfully using (and is successfully detecting renames and moves), it will now give rise to unnecessary copy operations for the new folder pair if you subsequently rename or move a folder.  No matter how much you delete from this folder structure and corresponding destination in order to simplify the structure, then create new folder pairs to test, it always shows unnecessary new files to copy. The only way to fix the problem is to actually run the operation rather than preview and from that point on, it successfully detects future renames.

    It appears the problem only starts when your SYNCTOY 2.1 folder pair is not the folder pair that created the contents of the destination folder in the first place. If you want synctoy 2.1 to work reliably, the destination folder must therefore initially be EMPTY.

    I really don't want to copy 184 gig all over again just to let synctoy start working.  It does appear to be happy with an existing destination folder when creating a new folder pair, so it should be intelligent enough to establish the folders are indeed in sync and not get confused when folders or files are renamed in the source.

    I'm still concerned that if this fundamental flaw has been overlooked then what other problems remain putting the contents of the destination folder at risk ?
    Tuesday, December 1, 2009 12:35 PM
  • We've established that Synctoy 2.1 will make duplicate copies in the destination of any files or folders that you rename when using the echo feature, if the destination folder already existed when the folder pair was first created.


    The following steps will recreate the problem:

    Choose a source folder and create an empty destination folder
    Install Synctoy 2.1 and create a folder pair with these two folders using ECHO
    Run the folder pair in order to populate the destination folder
    Delete the folder pair
    Create a new folder pair for the same source and destination folders using ECHO
    Run the folder pair and confirm 0 files to copy
    Rename a file and a subfolder, and move a subfolder within another subfolder
    Run the folder pair

    You will now observe the Synctoy 2.1 failed to recognise the file and folder renames, and the moved folder. It will just perform copy operations copying one files and the two subfolders with their new names to the destination, leaving the originals in place.  The same problem occurs if the destination had originally been created by another sychronisation method such as robocopy.


    Thus Synctoy 2.1 has some serious problems :
    1) A huge amount of time is wasted copying unnecessary data along with the wasted bandwidth
    2) If a renamed folder is very large, the time to re-copy may take longer than the time window available
    3) The original folder remains in the destination along with the new copy which will most certainly create confusion
    4) The destination folder is certainly no longer a true copy of the source


    The only way to avoid this problem is to completely delete the destination folder before creating a folder pair ! Thus, you can't delete a folder pair and create a new one in order to solve other problems that may occur. The moment you create a new folder pair using an existing previously synched destination, you will now not be be able to trust Synctoy 2.1 to recognise renames and moves.  

    Who knows, there may be other problems too. Much as I was looking forward to Synctoy 2.1 having found the previous versions didn't work, I am still unable to use V2.1 for regular backups of any folder stuctures, since I do not want to have to delete all my destination folders before creating a new folder pair.

    It appears that Synctoy 2.1 needs an update.
    Can someone from the Synctoy team please advise if this will be fixed in V2.2 and if you have any timescales?
    Thanks
    Tim
    Sunday, December 6, 2009 11:23 AM
  • Has anyone repeated the above steps to recreate the major problem in Synctoy 2.1 ?
    Has anyone observed this problem with their own folder pairs (i.e. a where you created a folder pair for an existing source and destination folder, rather than starting off with an empty destination folder?)
    thanks
    Tim
    Saturday, December 12, 2009 10:24 AM
  • In the right folder there are hidden synctoy database files (in my case only one or two in the main parent right folder). If you delete these and re-run the synctoy backup (try preview first) you should find it now renames and deletes in the right folder correctly.

    Cheers,
    Colin

    Saturday, December 12, 2009 9:15 PM
  • Hi Colin, thanks for the reply

    On the basis of your recommendation, I ran the main test again but I have concluded that synctoy still fails in the way I described, performing tens of thousands of unnecessary copies.

    Using Vista SP2  x86 2.5ghz Quad core pc with 3 gig ram :

    I prepared my source and destination folder, and confirmed they are fully synchronised by using ROBOCOPY with the /MIR switch. The folder contains 193.3 gigabytes (244847 files, 9723 folders).  The source is an local SATA drive (NTFS) and the destination is on an external USB drive (NTFS).

    I removed attributes from synctoy*.dat on both source and destination folders, then deleted synctoy*.dat
    I also removed attributes from synctoy*.dat within %userprofile%\appdata\local\microsoft\synctoy\2.0 then deleted synctoy*.dat

    Using Synctoy 2.1 I then created a new folder pair using echo and standard options.

    I ran preview which successfully ran in 26 mins 36 seconds, whicih successfully found the expected number of files and folders with no actions to perform.

    To perform the test, I renamed a single subfolder within the source, which contains 11.7g (56,563 files, 600 folders)

    Synctoy 2.1 exhibited exactlty the same major problem, finding 56,565 files to copy, nothing to delete or rename and 601 folders to create. (57,166 actions to perform)

    SYNC: 12/13/2009 20:54:40:056: Preview of datatestfolder (d:\data1\, f:\data1\) in time 00:34:14:170.
    SyncToy action was 'Echo'
    Found 57,166 actions to perform.
    Found 433,132 files that did not require action.
    Analyzed 210.9 files per second.
    Avoided copying 392,728,596,338 bytes in 433,132 files.
    Saved approximately 02:10:57:00 by not copying any files.


    Note - I did also perform the test using a smaller source and destination pair (631meg, 631 files, 125 folders) and renamed a folder within containing about a few files.  (As before, the destination already existed and was already syncrhonised when I originally created the folder pair).  On this occasion, Synctoy 2.1 worked and only performed renames and moves, avoiding any copies.

    It therefore appears that you may not replicate the problem if you rename a particularly small subfolder; I don't know how large the subfolder needs to be before Synctoy goes wrong.  I am pretty sure this won't be fixed without an upgrade so I'd certainly recommend avoiding Synctoy V2.1, though at this time there doesn't appear to be any mention of a proposed upgrade on this forum.

    If you have replicated this particular fault, please add the details of your test here including folder sizes  - I can't be the only one !  It was the first test I did when Synctoy 2.1 came out and I was disappointed to find it didn't work.
    Sunday, December 13, 2009 1:41 PM
  • I'm surprised that MS haven't responded to this problem report, whether to refute it or to confirm it.

    Don't you think that a synchronisation product that can result in the destination being different from the source (without showing that any errors occurred), is a product best avoided until fixed ?  Should you leave users with a serious bug in Synctoy 2.1 that puts their data at risk without acknowledging the fault or putting out a rapid update, whether or not it is free? Could you at least engage in discussion in this forum ?

    When your destination ends up with a renamed folder and keeps the original folder as well, isn't that going to cause confusion for anyone relying on this synched backup ?   If this happens on a large subfolder, the destination folder may be full long before the source, and then the backup will certainly fail. 

    Ping,  Joey Lang, Ji Guang, Nina, Deepa, Yunwen - we look forward to a response.
    Many thanks
    Tim
    Wednesday, December 16, 2009 11:07 PM
  • I did tests like the ones described above. goes wrong for me too in the same way.
    Synctoy 2.1 is unreliable, definitely not safe to use !
    If you want to be sure your destination folder is a proper backup without extra folders hanging around, you'll have to use something else. You think synctoy 2.1 worked, but when you check carefully, it screwed up if you renamed folder.
    would be interesting to see ms response to the problems reported here. like are you fixing it?!!!
    so long, hope v3 works.
    Tuesday, December 22, 2009 3:43 AM
  • Hi Tim -

    Thanks for testing SyncToy so extensively. I am looking at your last post and the scenario that you tested ( so feel free to let me know if there is a different scenario that does not work as well). Pasting an excerpt from your last post below -

    "Using Synctoy 2.1 I then created a new folder pair using echo and standard options.

    I ran preview which successfully ran in 26 mins 36 seconds, whicih successfully found the expected number of files and folders with no actions to perform."

    You only ran a preview - you did not actually do the Sync. And this does make a difference. A preview does not actually do any part of the sync ( including updating the metadata which would let us reconcile information that we know about the two folders on both sides). The number of files in a folder or the total size makes no difference to SyncToy ( except from a performance perspective of course). So doing that first sync is critical to us being able to detect future moves/renames.


    Thanks
    Deepa

    p.s. A large part of the team has been on vacation hence the delay in responses.
    Deepa ( Microsoft Sync Framework)
    Monday, December 28, 2009 10:29 PM
  • A:hover{text-decoration:underline;}A:link{color:#004E82;text-decoration:none;}A:visited{color:#004E82;text-decoration:none;}
    Hi Deepa.
    Thanks for the reply
    I did indeed assume that running the preview was enough to let Synctoy build its metadata to reflect the contents of the folders which is why I didn't proceed with running on some of the later tests.  (After all, having discovered there are 0 files to copy, it wasn't obvious that you must then click run)

    That would certainly explain why some of my later tests consistently failed even on smaller folders, and also explain why it then worked well after I actually ran the echo synchronisation job. In fact I did write
    "The only way to fix the problem is to actually run the operation rather than preview and from that point on, it successfully detects future renames."
     
    However, on the original large test, I definitely did run the sync operation before proceeding to the rename subfolder test and I still experienced the problems described.
     
    So I'll run the main test again, this time making absolutely sure I that after running the first preview that I run the sync operation. Then I'll proceed to test the rename of a subfolder in the source.
     
    Source and destination folder, already synched using robocopy, contains 189g (244,910 files, 9248 folders)
    Created Synctoy 2.1 folder pair using Echo.
    The Synctoy 2.1 preview took 27 mins, 6.7 secs and showed 0 actions to perform.
    The run took 35 mins 54.3 secs - copied 0 bytes in 0 files.
     
    Rename a subfolder within the source folder (12.1g, 56,574 files, 601 folders)
    Now when running Synctoy 2.1 echo sync on this folder pair, we should expect to see no deletions or new files,  just renames.
    However, the results are:
     
    Delete Folder: 602
    Delete :  18,973
    Overwrite: 0
    Rename: 37,601
    New: 18,973
    Create Folder : 602
     
    I notice this is exactly the same number of new files to copy as when I ran this test earlier.
     
    I'll look carefully through the list of files to copy in the preview and break this down into file sizes (this took a lot of scrolling and counting!)
     
    > 100,000,000 bytes : 2 files
    > 50m bytes, < 100m bytes:  4 files
    > 10m bytes  < 50m bytes:    36 files
    > 5m bytes < 10m bytes    : 43 files
    > 1m bytes < 5m bytes :152 files
    < 1m bytes > 500,000 bytes  :41 files
    < 500,000 bytes > 100,000 bytes : 273 files
    < 100,000 bytes > 50,000 bytes : 269 files (about 15% copied rather than renamed)
    < 50,000 > 10,000 bytes : 426 files (about 20% copied rather than renamed)
    < 10,000 bytes > 5000 bytes : 828 files  (about 6% copied rather than renamed)
    < 5000 bytes > 1000 bytes : approx 2050 files (about 50% copied rather than renamed)
    < 1000 bytes > 500 bytes : approx 11,400 files (about 80% copied rather than renamed)
    < 500 bytes > 100 bytes : approx 2700 files (about 70% copied rather than renamed)
    < 100 bytes        approx 750 files    (about 50% of files copied rather than renamed)
     
    I've checked the properties of many of these files, both large and small, and I can't find anything interesting features... some are ini files, cab files, exe files, some are small files with no file extension... but I really can't see what would cause Synctoy to copy rather than rename these files.
     
    Regards
    Tim
    Tuesday, December 29, 2009 5:35 AM
  • Hi Tim -

    Thanks for taking the time to test SyncToy 2.1. We use a heuristic to determine a rename as opposed to a delete/new of a file. The heuristic unfortunately is exactly that - a heuristic and if it fails to beyond doubt identify a rename, we then tag it as a delete/new of a file because mistakenly identifying a rename would result in data being lost. The current behaviour while not best with respect to performance will not result in data being lost. The only reason that you see this behaviour on a large dataset is because it just makes it more likely that we are unable to accurately identify a rename.

    We appreciate the time and effort you have taken to identify this problem. For a future release we will attempt to see if we can identify a rename more accurately.

    Thanks
    Deepa

    Deepa ( Microsoft Sync Framework)
    Wednesday, December 30, 2009 10:30 PM
  • Am relatively new to Synchtoy (started with 2.1), but as I understood the instructions, when you change versions of Synchtoy, you must first synchronize your folders, then change versions, then (without making any changes) synchronize with the new version. In other words, you're setting a synchronized baseline with the old version, and then showing the new version the baseline it's supposed to start from. The description in your first paragraph reads as if you made changes (deleted files) before synchronizing the first time with the new version. If that's the case, I wouldn't expect it to work, because the new version would be assuming what it finds the first time is already synchronized. To a program, that could be confusing.

    Once you changed the baseline without running the new version for the first time, you probably should have deleted the Synchtoy data file, essentially starting from scratch (see sleepycol's Saturday, December 12, 2009 9:15 PM below) .
    Friday, January 1, 2010 2:39 PM
  • I proposed this as an answer to another user having problems with SyncToy. Not exactly the  same as yours, but I still think worth a try.

    You say in  your first post that  "The following tests are using the 'Echo' feature of Synctoy 2.1."

    SyncToy has a bug, that if you set a folder pair to Echo when you first set it up, it doesn't echo properly. For many if not all users it does not pick up moves and deletions in the left folder, which means that junk accumulates in the right folder, making it useless for backing up.

    I would recommend that you (i) delete your folder pair, (ii) re-establish the folder pair and set to Synchronize before running it, (iii) after running change the action to Echo, (iv) make a change in the left folder, run ST and check it is working. Probably too simple for a complex problem like yours!

    Some users including you report other misbehaviour, like not picking up renames, or wanting to copy files L>R when not needed. Nonetheless I would try the sequence suggested above. If that doesn't work and Robocopy doesn't do the job, try FreeFileSync, which I am now using, as the last Patch Tuesday seems to have caused ST not to be able to load. (see http://social.microsoft.com/Forums/en-US/synctoy/thread/1c291737-080e-4bf8-8c05-ee4394979624#1c291737-080e-4bf8-8c05-ee4394979624)
    Thursday, February 11, 2010 10:02 PM
  • http://www.microsoft.com/prophoto/downloads/synctoybeta.aspx

    http://blogs.msdn.com/sync/archive/2008/08/14/now-available-synctoy-2-0-file-synchronization.aspx

    http://social.microsoft.com/Forums/en-US/synctoy/threads

    hi .

    some links that could be usefull

    have a nice day




    Scan with OneCare + 50 Windows 7even Tips + Plagued by the Privacy Center? REMOVE IT + Threat Research & Response Blog + Sysinternals Live tools + TRANSLATOR+ Photosynth + Microsoft Security + Microsoft SUPPORT + PIVOT from Live Labs + Microsoft Live Labs + Office 2010 beta + Get Windows LIVE!
    • Edited by Dabur972 Friday, February 12, 2010 12:12 AM link
    Friday, February 12, 2010 12:10 AM
  • Hi,

     

    ST 2.1 with a fresh sync couple between 2 drives, one on USB and the second on network (NAS). After 1H45 of work, ST decides that hundreds of files are new even if they are NOT new (perfectly existing). So I get this : 1995 failures !

    I was very happy with the ST 1.4 and I think to regress... :-(

    SYNC: 06/27/2010 13:04:46:906: SyncToy run of O vers Z NAS (O:\, Z:\) completed at 27/06/2010 13:04:46.
    SyncToy action was 'Echo'.
    SyncToy options were:
        Active for run all
        All files included
        No files excluded
        Do not check file contents
        Include read-only files
        Include hidden files
        Do not include system files
        Do not backup older files
        Some subfolders included
    SyncToy run took 01:43:39:984.
    Copied 26 369 194 090 bytes in 10 915 files in 01:43:39:984.
    Bytes per second 4 239 430,9, files per second 1,8.
    Avoided copying 520 270 128 145 bytes in 232 275 files that did not require action.
    Saved approximately 10:05:21:690 by not copying all files.
    Warning: 1995 failures occured.
    You can retry by selecting "Run" again or select "Preview" to see
    the operations that remain to be performed.

    SYNC: 06/27/2010 13:04:46:921: *** Error: Cannot write to the destination file. Impossible de créer un fichier déjà existant. (Exception de HRESULT : 0x800700B7) Copying O:\PC ENFANTS\conso électricité.xls to Z:\PC ENFANTS\conso électricité.xls

    Error: Cannot write to the destination file. Impossible de créer un fichier déjà existant. (Exception de HRESULT : 0x800700B7) Copying O:\Ziques\MIDI\mazurka à Léa.mid to Z:\Ziques\MIDI\mazurka à Léa.mid

    Error: Cannot write to the destination file. Impossible de créer un fichier déjà existant. (Exception de HRESULT : 0x800700B7) Copying O:\Mémoire IDPE\Ing.automatismeDMi01038.doc to Z:\Mémoire IDPE\Ing.automatismeDMi01038.doc

    Error: Cannot write to the destination file. Impossible de créer un fichier déjà existant. (Exception de HRESULT : 0x800700B7) Copying O:\Mémoire IDPE\J.doc to Z:\Mémoire IDPE\J.doc

    Error: Cannot write to the destination file. Impossible de créer un fichier déjà existant. (Exception de HRESULT : 0x800700B7) Copying O:\Mémoire IDPE\orgaDMi.xls to Z:\Mémoire IDPE\orgaDMi.xls

    Error: Cannot write to the destination file. Impossible de créer un fichier déjà existant. (Exception de HRESULT : 0x800700B7) Copying O:\Mémoire IDPE\visionorch.wmv to Z:\Mémoire IDPE\visionorch.wmv


    Schneider Electric Industries France
    Sunday, June 27, 2010 8:40 PM
  • How about giving us a reghack to disable the heuristic detection.  It's quite annoying.  Instead of having older files overwritten by newer ones, I'm ending up with duplicate files with a filename.1.ext.  It never used to do this garbage.  Maybe it's the latest .Net framework changes, not the heuristic detection, I'm not sure, all I know is it now wants to remane and then copy new everything instead of overwrite.  And it doesn't matter if I choose Sync or echo, doesn't matter if I delete the SyncToy_*.dat files or not.  I can even delete the entire folder from side 2 and then copy the folder over from side 1 and then when I run Synctoy it tells me they don't match and will do the rename and new causing duplicates.  Makes this product useless for me.  Can anyone recommend a freeware utility that does work?
    Tuesday, June 29, 2010 5:55 AM
  • I would propose the excellent, simple and straight SyncBack at http://www.2brightsparks.com/syncback/syncback-hub.html.

    It does in 10 minutes what SyncToy does not do in 1H45. Filters, preview... very simple.


    Schneider Electric Industries France
    Tuesday, June 29, 2010 12:46 PM
  • I appreciate your SYNCBACK recommendation and I went ahead and gave the freeware version a test.  I like the ability to schedule the sync but it has some drawbacks that unfortunately do not fit my needs.  SYNCBACK does not appear to maintain historical file structure information, this means that it doesn't know, since the last run, if a file is new on the right or was deleted on the left, so the best I can do with this is set it to ask whether it should delete or copy, which makes scheduling the task no longer a possibility.  Because of this same shortcoming it does not recognize when a file or even worse an entire folder has been renamed and will then copy the old name from the destination and the new name to the destination making duplication problems as bad or worse than SyncToy 2.1.  Also, and this may just be a setting I could change, but it made me verify each and every copy.
    Wednesday, June 30, 2010 5:57 AM
  • Hello Tim,

    I appreciate the contributions you made to this forum on Synchtoy. They certainly helped me to give Synchtoy a try.

    The download and install were easy. No problem there.

    I then paired my C:\ with G:\ which was a 2 TB external WD drive. I was not happy with WD Smartware so that is why I was looking for something very simple to just give me what I wanted.

    I used 'synch' and it started a 8:30 am. It did a first pass by 8:23 am.  It then started to copy my C:\ drive files. It took about 5 hours total (finished at 13:30pm) It showed that it had copied 41.6Gig. I am not sure if this is good or bad since I have an older 32 bit machine. 

    Operations           Successful     Failed       Total

    New                      162,596            229       162,825

    Create Folder          12,207               2         12,209

    All Operations        174,804           231        175,035     

    The files that failed mostly appeared to be in use.

    I looked at the G:\ Drive and it looks exactly like my C:\. All my data and programs seem to be there and accessible. 

    If I get any errors during my next 'synch' I will post it.

    Thanks for all your testing.

    AC

     

     

    Saturday, July 3, 2010 1:40 AM
  • ac603>231 failed copy is not good. Some files are missing.

    trainableMan> I have regressed to ST 1.4. Analysis done in 5 minutes, successful copy in 5 more minutes (instead of 2 hours then failure)

     

    Before coming back to ST 1.4, I have tried a last time ST 2.1 between my two 500Go USB HD (NTFS). I had the same diagnostic than between USB drive and NAS : very long analysis and hundreds of files tagged to "new" despite they already exist.


    Schneider Electric Industries France
    Saturday, July 3, 2010 9:09 AM
  • Yes, I tried 2.1

    I have a dual boot system and wanted to duplicate the SyncToy actions on Win7 and XP SP3 (which has had V 1.4 for years).

    When I tried to Sync folders using 2.1, I noticed that it wanted to overwrite files that did not need to be. However, I'm still trying to figure it out, as the "newer" files were all one hour younger than the "older" files. Something to do with daylight saving no doubt (I'm in the Southern Hemisphere)

    I also got a few of the error messages about Synctoy metadata.

    Anyway maybe one day when I have a spare weekend, I will try to find out what's going on, but for now I am going to live with 1.4 and its foibles... It works without issues on both OSs.

    Cheers,

     

    • Proposed as answer by coolsailor Thursday, December 5, 2013 9:11 PM
    • Unproposed as answer by coolsailor Thursday, December 5, 2013 9:11 PM
    Tuesday, November 30, 2010 9:18 AM
  • 2013/12/05 - I realize this is an old topic but here is a possible answer if anyone is interested. Since I still had the rename problem I tried this and it worked, using 2.1.

    SyncToy will process folder renames correctly if the pair was originally set up using SyncToy. That means SyncToy must be used to create the new destination top level folder and do the original copying to create the sub folders/files, obviously by running the first sync or echo. I tried several tests on small folders and this is what happened. It would no doubt be a huge problem on large folders.

    Test1. Created back up copy of folder/files using file explorer (win8), created folder pair in SyncToy and ran first sync. All ok, then I renamed a sub folder and then SyncToy wants to recopy all affected folders/files, creating another set with identical info under the 'new' renamed sub folder.

    Test2. Created new pair with SyncToy and let it create the new destination folder, ran first sync, it copied all folders/files, all ok. Renamed one sub folder and then ran a sync, all still ok, sub folder correctly renamed on opposite side and files renamed to match, no recopying needed.

    Summary - SyncToy must do all the initial legwork to be able to recognize folder renames. Still will not recognize top level folder rename though. SyncToy will work ok with 'existing' folder pairs but will not recognize folder renaming, unless you rename the same folder on both sides.

    Obviously it is pathetic if it cannot recognize renames because using SyncToy to do the original folder/file copying is just way too slow.

    Thursday, December 5, 2013 9:36 PM