I assume your using it through the main application. I only say this b/c I do the same thing but I use a task to do it hourly w/ the cmd interface. I know when I used to backup my computer using the main application it would freeze the application and maybe
slow the computer, but never enough to make it unusable (I might be remembering wrong). The sync would eventually fail b/c files system files would change and then not be valid for sync.
When you delete the file it doesn't have to compare anymore so its faster. I know most of this really doesn't answer the question, although it might help to sync the entire thing more frequently so it has less to check each time.
When it fails does it give an error message or do you force close it?
I'm also on Win7 x64, SyncToy 2.1. Sorry I couldn't be more helpful.