none
Robocopy with Multi-threading switch RRS feed

  • General discussion

  • Its desired to copy a folder containing many files quickly (10 GB) between 2 machines within a network. While trying to leverage on the inbuilt multi-threaded copy feature, which is supported by robocopy, following are the observations in our case -

    Say,

    The local host machine (A), which contains the source folder, & also which Robo-copy is executed.The destination machine (B) on the network, which is also on the same subnet as (A), the place where its desired to copy to.

    The command line is of the form: robocopy <Source_path> \\<IP of B>\<drive>$\<Destination_path> /MT /E /R:10

    The problem is as follows -

    When copying across the network from A to B, the multi-threaded Robocopy, somehow does not yield an advantage. Yet however, everything remaining the same, when trying to copy within the same machine i.e A to A, MT option did work and saved considerable time.

    Is this problem inherent to robocopy. Has anyone come across similar observations, especially while copying across machines in a network, either in the same LAN or elsewhere ? If so, how can they be overcome ?

    Would robocopy [MT] fail to give an advantage over  single threaded Robocopy when dealing with NetApp file storage ?

    Pl. let me know.

    Thanks,

    venky

    • Changed type Bill_Stewart Tuesday, November 7, 2017 10:22 PM
    • Moved by Bill_Stewart Tuesday, November 7, 2017 10:24 PM This is not "teach me how to use robocopy" forum
    Wednesday, September 27, 2017 2:54 PM

All replies

  • This is not a robocopy Q&A forum or a storage/network/performance Q&A forum.

    Sorry, I don't know a better place to ask. I would recommend searching for a more appropriate forum to ask your question.


    -- Bill Stewart [Bill_Stewart]

    Wednesday, September 27, 2017 2:55 PM
  • Shares do not necessarily allow multiple concurrent operations from a remote user. 

    \_(ツ)_/

    Wednesday, September 27, 2017 5:50 PM
  • I am working from last 2 years on migration of File share to file servers or on NAS.

    First thing I recommend is to share the folder over network rather than using whole default drive. Second thing is to use correct switches in your robocopy command .

    Go through the below link for getting more on robocopy.

    https://technet.microsoft.com/en-us/library/cc733145%28v=ws.11%29.aspx?f=255&MSPPError=-2147217396

    We are doing migrations of data as large as 50 -60 TB using robocopy. And we are not facing any issue, the only thing it will do is it will acquire lot of your bandwidth of network.

    Thursday, September 28, 2017 1:16 PM
  • Couple things to  look at. 

    1. You need to define a number with your MT.  Otherwise it accepts the default of 8, which will give you little benefit.  I've seen improvement at 32, and found the sweet spot between speed and CPU use at 64.  128 will require a lot of CPU resource.

    2. If you've done 1 already, is there network optimization/conditioning such as a steelehead between server A and B.  Is there bandwidth considerations? i.e. remote site you're sending/receiving. 

    3. Files being locked?  You have a retry counter of 10.  If there are permissions preventing access files might be getting hung.

    4.  File count.  If it's large chunks of data i.e. 4 large multi GB files, you're not going to see a lot of improvement.  the MT switch assists with parsing/comparing/transferring of large number.  The trade off is increased CPU overhead. If there is a lot of small files, you would see the most benefit.

    5.Reference a common one I use for moving data between locations.  I use this specifically for directories with high file counts. It also give you a log file that will show file locks, permission issues etc..  Also run it from your source server if possible.   

     Robocopy Source  \\Destination *.* /S /COPYALL /MIR /B /R:5 /W:3 /MT:64 /NP /TEE /LOG:C:\Temp\T_Drive_Robocopy.log


    Thursday, September 28, 2017 2:19 PM
  • Thus why you set the copy from the source location locally, and let it take CPU overhead. Local source to a remote destination is fine. I've done hundreds of migrations using /MT:n while comparing the transfer speed without using it, and there is significant variance.
    Thursday, September 28, 2017 2:24 PM