locked
file transfer speed over GB LAN RRS feed

  • Question

  •  

    I have an XP MCE machine and a WHS machine connected to my GB LAN on either side of a Netgear GB switch. When I first installed the switch (replacing an old 100 MB switch), I got susstained file transfer rates of 300-400 MB/s from MCE to WHS. Now when I transfer, for example, recorded TV files from MCE to WHS the transfer starts off fast, say 400MB/s for 5 or 10 seconds then slows to nothing for several seconds then starts up again more slowly. Typically it will finally settle at around 100 MB/s pausing occasionally during transfer.

     

    Can anyone axplain why file transfer should have slowed down from an initial susstained rate of 300-400 MB/s to a slower, intermmittant rate? Is there anything I can do to optimise file transfer speeds.

     

    BTW all my HDDs are SATA II. I have 1GB or RAM in both the MCE and WHS boxes.

     

    Thanks

     

    Ben

    Wednesday, October 24, 2007 6:26 AM

All replies

  • Hi Ben

     

    Have you installed anything else on your WHS box, for example Antivirus?

     

    Andrew

     

    Wednesday, October 24, 2007 6:34 AM
    Moderator
  • Andrew,

     

    Good point. I have ClamWin installed but this only scans on a scehdule. It doesn't scan files on access.

     

    I have got uTorrent installed as well, but this is currently inactive - no downloads running and no files being shared.

     

    Ben

     

    Wednesday, October 24, 2007 6:40 AM
  • Andrew,

     

    I have tried watching hte processor activity on the WHS box before and during a file transfer. I get this erratic transfer speed even if WHS is doing nothing else.

     

    Ben

     

    Wednesday, October 24, 2007 6:41 AM
  • Hi Ben

     

    Do you have another machine you can try copying the files from the MCE machine to, just to rule out "network" issues?


    Andrew

    Wednesday, October 24, 2007 6:47 AM
    Moderator
  •  

    I've raised this issue a number of times, without ever receiving a satisfactory answer.  I see exactly the same issues, with a number of different PC's trying to transfer files.

     

    It really does seem that WHS kills network transfer speeds, to the point where I'm tempted to quit it.  My network regularly sees sustained 60mb/s when transferring between other PCs, but typically 1-8mb/s (and often a LOT lower) with some real fluctuation (sometimes even coming to a halt) when transferring to or from WHS.

     

    So far, it certainly seems that it's the balancing funciton that's causing the problems - WHS machines with one drive in do not appear to exhibit the issue from anecdotal evidence. 

     

    I've observed this on two completely different hardware platforms, with a couple of the beta releases and now with OEM.

    Wednesday, October 24, 2007 12:33 PM
  • Ben,

     

    You may find this information in some other posts as I know I read about it on this forum though it is a workaround not a solution.

     

    If you are transferring a large file or a group of files then turning off duplication while you copy the files will result in better throughput. Once you have transferred the files you can turn duplication back on.

     

    This worked for me though I was still disappointed with the WHS throughput on my Gb LAN and suspect that there are some more reasons why WHS is relatively slow.

     

    Robert

    Wednesday, October 24, 2007 1:32 PM
  • Hi Robert,

     

    Are you suggesting I turn off duplication on all shares or just the share I am copying to? If you mean all shares (I have several hundered GB or duplicated data), wouldn't it take a lot of time to re-duplicate afterwards? If you mean just the share that data is being transferred too then I do not duplicate this share anyway.

     

    Cheers

     

    Ben

     

    For Info, a picture from task manager showing my file transfer rate can be seen here.
    Wednesday, October 24, 2007 5:04 PM
  • Ben,

     

    I was talking about turning off duplication only on the folder you are transferring too, so if that is an unduplicated folder then I don't have any other suggestions for you, except using jumbo frames if all your gear (NICs and switches) is capable of it.

     

    Robert

    Wednesday, October 24, 2007 5:34 PM
  • Also, check the NIC drivers, there have been quite a few instances where it's the network card that causes the bottleneck - even on gigabit networks. Also, the settings, 'Full Duplex' 'Auto-Negotiate' etc. Try altering these to fixed speeds.

     

    Colin

    Wednesday, October 24, 2007 8:07 PM
  • The problem with any suggestion about network setup or drivers is that speeds are fine before WHS starts balancing.  Single disc with WHS is FAST - the same as transfers to other machines in the network.  However, once you have two or more drives in the pool then transfer rates CRASH...

     

    Surely this cannot be a setup/hardware/driver issue in these circumstances?

    Thursday, October 25, 2007 8:22 AM
  • I'm pretty sure that there's a combination of factors playing into this particular issue. Some are client-side, some are network or driver configuration, and some are Drive Extender. I have this problem (to an extent) myself; on a gigabit network a large file will start to transfer at 20-40 MiB/s, then will rapidly slow down to under 10. At that point it will cycle from 10-20, with occasional moments when it drops to near 0. Average is a bit under 10 MiB/s with files in the GiB range. I haven't worried too much about it, frankly, because it only affects me occasionally. I normally only copy huge files (DVD or CD-ROM .isos) overnight anyway.
    Thursday, October 25, 2007 2:51 PM
    Moderator
  •  

    I posted a similar question in another forum here. This is getting really aggravating! I do not have duplication turned on at all. Something has to be done about this, this is what the OS was intended for, large file transfers. Gotta be something that can be done or have the engineers have a better look into the code or something.
    Sunday, November 4, 2007 1:42 AM
  • I'm beginning to wonder whether a slowdown happens when the MFT is suffering from fragmentation. Plus, I have been reading and article, part of which said

     

    "The phenomenon of throughput slowdown was traced to memory management of disk writing functions in the windows operating system, which were not optimized for a sustained use without allowance for periodic flushing of cached data."

    Part of cause of the problem, appears to be the standard Windows File I/O function. Bypassing this to access the Win32 API directly showed significant performance improvements for large files.

    "The amount of time taken to write the buffer contents to the hard disk varies significantly. Most of the time,the disk write process takes a small mount of time; often less than a millisecond.However,the operating system is occasionally burdened with overhead administration duties and cannot immediately service the write request.
    When this happens,individual write times may increase by several orders of magnitude. Both peak disk write times and overall throughput are significant indicators of system performance. Peak disk write time determines the amount of cache buffer memory required to temporarily store data while the operating system is occupied with disk writing and other administrative tasks."

     

    This was in relation to an NTFS file system on W2k3. Sound familiar?

     

    Colin

    Sunday, November 4, 2007 3:42 PM
  • Perhaps the following might provide some solution:

     

      20 Feb 2007, 7:27 PM UTC
    T. Headrick

    Moderator


    Posts 282
    Answer Re: Slow transfer speed when copying large files to WHS
    Was this post helpful ?

    How big is the first hard drive in your home server?  Windows Home Server uses 10 GB of the primary hard drive for the operating system and the remainder is used for the first DATA volume.   This first DATA volume is used for copying data from PCs to your home server and then Drive Extender will move it to one of your other hard drives. 

    Also, if Folder Duplication is turned on for one or more of the given shared folders then Drive Extender will make sure there are 2 copies of each file stored on 2 separate hard drives.   And lastly, if you have enable "Media Library Sharing" on the Photos, Music and/or Videos folder, then Windows Home Server will start indexing the media as it is added to these folders.

    If you want to copy lots of data to your home server - here are a few recommendations:

    • Have a large primary hard drive - I have a 300 GB drive in my home server
    • Turn off 'Folder Duplication' on a shared folder during a big copy operation and turn it back on after the copy completes
    • Turn off "Media Library Sharing" on a shared folder and turn in back on after the copy completes.

     

     




    Windows Home Server Team

       Report Abuse  

    Sunday, November 4, 2007 3:54 PM
  • Also I found the following site very helpful in understanding bandwidth speeds and conversion tables:

     

    http://web.forret.com/tools/bandwidth.asp

    Sunday, November 4, 2007 6:23 PM
  • I'd been messing around with this over the last couple of days... what I found was when two drives on whs, it would take longer to move a 5 gb file than if just one drive.  If drive duplication was on the folder you were moving it would take longer than a folder without duplication.  From multiple tests (not very good tests) I found that one drive = 2:45 (minutes/seconds) to 3 minutes, two drives with folder not duplicated = about 5 minutes, and two drives with folder duplication = 7 to 10 minutes.  All using 5 gb files. 

     

    One thing that I note, is that when testing the single drive, I found that some of my test folders took up to 5 minutes, while some took 2:45 to 3 minutes. After looking at the folders to see what was different, I found that all the folders that took 5 minutes had one thing in common... they had sub-folders (ie folder/subfolder/ then data files), while folders that took 2:45 to 3 only had the one folder (ie folder/ then data files).

     

    I picked 5 gb since my issues seem to start at about 4.5 gb.  If I move a file (or folder with files) 4.5 gb or less, I see no issues with performance.  Over 4.5, performance drops off considerably at the end of the move...

     

    My observations, of course, you would really need to test 30 or more whs systems and maybe have 90 computers to see if my issues are the same across the board. 

     

    Sunday, November 4, 2007 7:55 PM
  • i'm having the same problem... over wired 100mbit network... only 3mb/s.... no duplicate folders nor media sharing active ...
    any solution yet ?

    best regards
    Friday, November 9, 2007 9:03 PM
  • I have the exact same problem. I got the HP MediaSmart server. When it had just the one drive in it, I could transfer a 2 gig file in under 30 seconds. It had the same transfer speed as when I transfered the same files to a Vista machine sitting right next to the server on the same switch.

     

    As soon as I added the second hard drive... bam! Transfers fast for the first few seconds then looks like it stops then trickles up to something horrifically slow. This only happens with 250 meg+ files. It seems to be just fine below that. I can have 5-10 transfers going at one time and sustain a 20-30 MB/s transfer speed (I use DU Meter to monitor the speed).

     

    Turning on duplication for the folders didn't slow anything down. It didn't matter if the second drive was internal or external. I thought it was a USB thing, so I bought a new internal SATA II 3GBs drive, and it is still slow.

     

    Does the OS need a couple gigs of RAM to buffer things better? Transferring files of any size from the server is just fine. It only seems to be transferring them to the server. This is only a big deal right now as I'm copying all my stuff from my various machines onto this machine.

     

    I'm glad to see I'm not the only one experiencing this EXTREMELY annoying problem.

     

    Bryan

     

    Thursday, November 29, 2007 4:09 AM
  •  

    I have exactly the same problem with copying large files to the server.
    The copy starts really fast but then slows down because half way the file transfer
    de demigrator starts copying the file from the landing zone to the other hard drives.

     

    So the system is accessing the system drive twice (reading and writing at the
    same time).... and that slows down the file transfer dramaticly.

     

    I think it would be better when the system waits till the end of the file transfer and then
    starts moving the files from the landing zone.

    Thursday, November 29, 2007 3:48 PM
  •  

    Okay, I'll jump on the train.

     

    I'm having the same issues as the rest of you.  I start copying a 10GB file from my Vista Home Premium x64 to \\WHS\Videos and it starts off like a raped ape and then slowly dies.  Sometimes it will get halfway and the transfer just stops...really strange.  I seem to get better results if I transfer directly to the D: drive as opposed to going to the share.

     

    I don't think my WHS is doing it's job with the balancing.  Take a look at the screenshot of my Disk Manager:

     

    http://www.lastoria.com/whs.jpg

     

    Shouldn't the percentage used by somewhat the same?  I don't understand why some drives are almost empty and others are barely used...doesn't seem logical.  Can a few other people take a screenshot of their Disk Manager to compare?

    Friday, November 30, 2007 11:51 PM
  • Tim, "balancing" is a misnomer, and I'm pretty sure the WHS team regrets choosing that particular term. Windows Home Server makes no attempt (that I can see) to actually balance the load across multiple drives. All it wants to do is keep D: clear, and make sure that files that should be duplicated are duplicated. It just so happens that those two functions normally result in a more or less even spread of files across your drives, most of the time. There's no imperative to do that though. If you add a new 500 GB drive to a server that has no files on D:, I would be surprised if anything moved onto that new drive without additional human intervention.

    So my advice is to ignore Disk Management, because it doesn't tell you anything useful.
    Saturday, December 1, 2007 4:39 AM
    Moderator
  •  BenHughes wrote:

     

    I have an XP MCE machine and a WHS machine connected to my GB LAN on either side of a Netgear GB switch. When I first installed the switch (replacing an old 100 MB switch), I got susstained file transfer rates of 300-400 MB/s from MCE to WHS. Now when I transfer, for example, recorded TV files from MCE to WHS the transfer starts off fast, say 400MB/s for 5 or 10 seconds then slows to nothing for several seconds then starts up again more slowly. Typically it will finally settle at around 100 MB/s pausing occasionally during transfer.

     

    Can anyone axplain why file transfer should have slowed down from an initial susstained rate of 300-400 MB/s to a slower, intermmittant rate? Is there anything I can do to optimise file transfer speeds.

     

    BTW all my HDDs are SATA II. I have 1GB or RAM in both the MCE and WHS boxes.

     

    Thanks

     

    Ben

     

    400MB a second?? please note that there is a difference between a  "B" and "b"

     

    If you have seagate HDD make sure the jumper is set to SATAII speeds, also check if perhaps you have jumbo frames enabled?

    Sunday, December 2, 2007 2:46 AM
    Moderator
  •  db298 wrote:
    The problem with any suggestion about network setup or drivers is that speeds are fine before WHS starts balancing.  Single disc with WHS is FAST - the same as transfers to other machines in the network.  However, once you have two or more drives in the pool then transfer rates CRASH...

     

    Surely this cannot be a setup/hardware/driver issue in these circumstances?

     

    I have seen this with my EX470.  When I first pulled it out of the box I created a share on the C drive just for speed tests and I was able to write up to 60 MB/s.  Ever since I added 3 more hard drives to it that same share on the C drive has never been the same.

     

    It is almost like the SATA controller on the mb is overloaded???

    Sunday, December 2, 2007 2:26 PM
  • I was having trouble with large files to my HP Mediasmart Home Server. I had been using a Netgear Gigabit switch. I needed that 5 port swtich elsewhere on my network, so I bought a new, bigger 8 port switch from D-Link to put in it's place. I now get around 30 MB/sec on my transfers to Windows Home Server for the first 2 gig of the file (4.7 gig in total), and then you can tell when the duplication effort kicks in and drops down to 19MB/s. It doese NOT however do like it did with the other switch where it dropped to nothing and then trickle back up to 8MB/sec over time.

     

    I am much happier with the system at this point.

     

    Bryan

    Monday, December 17, 2007 2:37 PM
  • Try transferring the same file in the other direction - from WHS to client PC.  If that is much faster then this must be an issue with the WHS balancing process slowing things down, is it not? 

     

    I find that I get much faster transfer speeds going from my WHS box than to my WHS box, at least over the PCs that are connected to my Gigabit switch and have Gigabit cards.

    Tuesday, December 18, 2007 10:58 PM
  • Check to make sure your files are not fragmented severly (I've run some file servers, and it's amazing what fragged files do to TPT once your number of file requests goes over 1).

     

    Also check to make sure that you don't have other files being pulled at the same time, which can cause HD thrashing if BOTH files are fragmented, even on SATA2.

     

    -Chad

    Thursday, December 20, 2007 9:43 PM
  • do you have Duplication turned on on your Shared folder?

    if so, at one point WHS will start replicating the file you are copying, and if you have only one HDD or are tight on disk space then that could have an impact.  BTW, I think you confused MB (mega-byte) with Mb (mega-bit).

    Tuesday, January 18, 2011 1:27 AM
  • you will also see a drop in performance when WHS is starting the backup process!

    review utilization of CPU, memory, network running Task Manager on WHS to find out more.

    Gigabit LAN certainly helps to overcome bottle-necks, and your WHS should have at least 2 MB RAM.

    Tuesday, January 18, 2011 1:29 AM