locked
Recently setup a new server and in the process of migrating the data from my old server hard drives to new storage pool transfers have become really, really slow through explorer? RRS feed

  • Question

  • Hi!

    I recently ran out of physical space in the case I was using for my server. I purchased a Norco 4020 (great case!) with the ability to hold over 20 hard drives. For a number or reasons I decided that when I transfered from my old case to my new case I was going to reinstall Home Server. For whatever reason server re-installation was not a possibility so I did a clean install with the idea of transferring my data from the DE\shares folders on all my hard drives to the newly created storage pool one at a time. So I started out with the OS installed on a 500GB drive and two 1.5TB storage drives. I was going through hard drives transferring the data with no problems at all.

    I previously had folder duplication turned on for every folder so I have been using WinMerge to merge the data from DE\shares on my old drives to the new storage pool. When all the information was transfered from the hard drive I would add it to the storage pool and move on to the next drive. Best I can tell WinMerge simply compares the contents of two folders and then tells explorer what folders and or files needs to be moved over. This worked great until about the 8th drive in. Now file transfers take FOREVER. It literally takes explorer 4 hours to transfer a 5GB file from one hard drive to the pool. Thinking WinMerge might be the culprit I selected a 5GB file and manually transfered it from one HDD to the storage pool with the same outcome.

    At this point I thought it may be a hardware issue. I ran HDD tach on each of my drives and they are all able to read and write about 80MB/sec. I then used a program called FastCopy to move the same 5GB file I was talking about eariler from a hard drive not located in the storage pool to the storage pool and this was my result...

     

    TotalRead = 4660.7 MB
    TotalWrite = 4660.7 MB
    TotalFiles = 1 (1)
    TotalTime= 82.13 sec
    TransRate= 56.75 MB/s
    FileRate = 0.01 files/s

     

    So as you can see this program was able to transfer the file as you would expect. This same transfer would take 4+ hours using explorer. This baffles me. I would continue to use this program but the problem is I really need to use WinMerge and therefore explorer to transfer my data as I have so many duplicates from having folder duplication turned on. 

    Any ideas? :(

    Sunday, April 25, 2010 4:44 PM

Answers

  • Just in case someone stumbles upon this in the future...

     

    The problem was a 500GB Seagate Hard drive. The drive is fine, it works on any computer with the exception of Home Server and only when transferring to the pool. As you can see in the graphs above.

    Who knows why it is like that, but I can tell you for sure it was a MASSIVE pain to figure it out. Multiple SATA controllers, multiple motherboards, installs, blah blah blah.

    If you're having the same problem I was, may God have mercy on your soul... 

    • Marked as answer by inXorable Saturday, May 22, 2010 2:23 PM
    Saturday, May 22, 2010 2:22 PM

All replies

  • Hi!

    I recently ran out of physical space in the case I was using for my server. I purchased a Norco 4020 (great case!) with the ability to hold over 20 hard drives. For a number or reasons I decided that when I transfered from my old case to my new case I was going to reinstall Home Server. For whatever reason server re-installation was not a possibility so I did a clean install with the idea of transferring my data from the DE\shares folders on all my hard drives to the newly created storage pool one at a time. So I started out with the OS installed on a 500GB drive and two 1.5TB storage drives. I was going through hard drives transferring the data with no problems at all.

    I previously had folder duplication turned on for every folder so I have been using WinMerge to merge the data from DE\shares on my old drives to the new storage pool. When all the information was transfered from the hard drive I would add it to the storage pool and move on to the next drive. Best I can tell WinMerge simply compares the contents of two folders and then tells explorer what folders and or files needs to be moved over. This worked great until about the 8th drive in. Now file transfers take FOREVER. It literally takes explorer 4 hours to transfer a 5GB file from one hard drive to the pool. Thinking WinMerge might be the culprit I selected a 5GB file and manually transfered it from one HDD to the storage pool with the same outcome.

    At this point I thought it may be a hardware issue. I ran HDD tach on each of my drives and they are all able to read and write about 80MB/sec. I then used a program called FastCopy to move the same 5GB file I was talking about eariler from a hard drive not located in the storage pool to the storage pool and this was my result...

     

    TotalRead = 4660.7 MB
    
    TotalWrite = 4660.7 MB
    
    TotalFiles = 1 (1)
    
    TotalTime= 82.13 sec
    
    TransRate= 56.75 MB/s
    
    FileRate = 0.01 files/s

     

    So as you can see this program was able to transfer the file as you would expect. This same transfer would take 4+ hours using explorer. This baffles me. I would continue to use this program but the problem is I really need to use WinMerge and therefore explorer to transfer my data as I have so many duplicates from having folder duplication turned on. 

    Any ideas? :(

    Have you logged into the server desktop and looked at Task Manager to see if any processes are running that might be bogging it down?  Perhaps WinMerge is doing something that's "going around" DE (I've never used it so I don't know if that's the case or not).
    Sunday, April 25, 2010 4:50 PM
    Moderator
  • When looking at Task Manager everything seems to be fine. Like I said I am pretty sure WinMerge simply tells explorer what needs to be copied over and doesn't use its own file copy method such as FastCopy. This was confirmed when I manually copied over a file outside of WinMerge using explorer and had the same problem. 
    Sunday, April 25, 2010 4:57 PM
  • When looking at Task Manager everything seems to be fine. Like I said I am pretty sure WinMerge simply tells explorer what needs to be copied over and doesn't use its own file copy method such as FastCopy. This was confirmed when I manually copied over a file outside of WinMerge using explorer and had the same problem. 
    Are you copying to the UNC path (\\server\[sharename]) or something else?
    Sunday, April 25, 2010 8:09 PM
    Moderator
  • I am copying from DE\Shares on the old drive to \\ServerName\Folder

    Example...

    DE\Shares\High Definition Movies to \\Home-server\High Definition Movies

    Sunday, April 25, 2010 8:25 PM
  • Instead of

    DE\Shares\High Definition Movies to \\Home-server\High Definition Movies

    shouldn't it be

    DE\Shares\High Definition Movies to \\Home-server\Shares\High Definition Movies


    --
    ______________
    BullDawg
    Associate Expert
    In God We Trust
    ______________
     
    "inXorable" <=?utf-8?B?aW5Yb3JhYmxl?=> wrote in message news:04ac4feb-bb49-4d90-9368-6f46ab850813...

    I am copying from DE\Shares on the old drive to \\ServerName\Folder

    Example...

    DE\Shares\High Definition Movies to \\Home-server\High Definition Movies


    BullDawg
    Monday, April 26, 2010 8:41 AM
  • Instead of

    DE\Shares\High Definition Movies to \\Home-server\High Definition Movies

    shouldn't it be

    DE\Shares\High Definition Movies to \\Home-server\ Shares\ High Definition Movies

    No. A share is a share, whether accessed locally or remotely.

    I'm not on the WHS team, I just post a lot. :)
    Monday, April 26, 2010 12:31 PM
    Moderator
  • Ok, I got it now.  I had  a senior moment and was mixing apples and oranges.  Sorry I added confusion to the mix.

    --
    ______________
    BullDawg
    Associate Expert
    In God We Trust
    ______________
     
    "Ken Warren" <=?utf-8?B?S2VuIFdhcnJlbg==?=> wrote in message news:30f7935c-fab3-4ef0-8c61-1b432ecaf112...
    Instead of

    DE\Shares\High Definition Movies to \\Home-server\High Definition Movies

    shouldn't it be

    DE\Shares\High Definition Movies to \\Home-server\ Shares\ High Definition Movies

    No. A share is a share, whether accessed locally or remotely.

    I'm not on the WHS team, I just post a lot. :)

    BullDawg
    Monday, April 26, 2010 5:02 PM
  • So...

    I thought maybe if I hooked the old HDD to my desktop computer and transfered the data though my network it might go faster (strange thought, I know) and basically when I started the transfer it started at around 80MB/sec and slowly went down to 3MB/sec and is still falling, the exact same thing it did via explorer.

    What is going on?!

    Monday, April 26, 2010 8:37 PM
  • So...

    I thought maybe if I hooked the old HDD to my desktop computer and transfered the data though my network it might go faster (strange thought, I know) and basically when I started the transfer it started at around 80MB/sec and slowly went down to 3MB/sec and is still falling, the exact same thing it did via explorer.

    What is going on?!

    Maybe that drive it's now trying to copy to is having issues.  Have you tried running chkdsk /r on each drive in your server?
    Tuesday, April 27, 2010 4:01 AM
    Moderator
  • This is happening with multiple drives. The speed is very slow for both a 1.5TB Seagate and 750GB Western Digital...
    Tuesday, April 27, 2010 2:31 PM
  • I took the time to run a drive balancer add-in as I saw a number of my hard drives were nearly 100% full with others having <5% usage even though I know this is standard for WHS. The add-in ran over night and balanced my drives but I am still having the same problem. Burst transfer speed in the 80MB/sec range quickly falling off to almost 0MB/sec transfer when sending anything to the pool. :(
    Wednesday, April 28, 2010 12:26 AM
  • Update:

    This problem only happens when I transfer to the shared folder in the pool. Example...

    If I transfer from...

    E:\DE\shares\High Definition Movies directly to D:\DE\shares\High Definition Movies then I have no problems at all.

    HOWEVER...

    If I transfer from

    E:\DE\shares\High Definition Movies to \\home-server\High Definition Movies then everything grinds to a halt.

    I believe if I do a direct transfer like I was saying above I am circumventing Drive Extender (which would also explain why FastCopy works) which would lead to problems in the future. Based on this do you guys think this is a program with Drive Extender and if so what should I do to further diagnose what is going on?

    Thank you for all your help!

    Wednesday, April 28, 2010 10:05 PM
  • Another update:

    I had run out of things I could think to try so I attempted a server reinstall but this failed. I then proceeded to reinstall the server from scratch on a new system drive and proceeded to attempt to transfer my data. The first hard drive went fine but it grinned to a halt on the second drive. 

    I cannot transfer something from E:\DE\shares to \\home-server\shares without it taking forever. I am simply dragging the file over when I attempt this.

    I can however, transfer something from E:\DE\shares to D:\DE\shares without any issue at all. 

    WHAT THE HECK IS GOING ON?! I am beyond frustrated.

    Saturday, May 1, 2010 3:56 AM
  • You were advised several days ago to run chkdsk on all the drives in your server . Have you done so?
    I'm not on the WHS team, I just post a lot. :)
    Saturday, May 1, 2010 5:14 AM
    Moderator
  • Yes. I only have three drives in the server now. The system drive (500GB Western Digital) with a fresh install of WHS, a 500GB Western Digital that I had no problems transferring to the pool and a 750GB Western Digital that is outside of the pool that I am trying to transfer to the pool. 
    Saturday, May 1, 2010 5:47 AM
  •  

    Here are some graphs to show you what I already knew was happening:

     

    This is write performance when sending a 11GB file to the pool. As you can see it stopped for nearly a minute at one point and when I ended the logging was not even 5% complete. Hosted by imgur.com This is write performance when sending that same file directly to the D: drive. Notice the scale on the left. In this graph the file transfer completed successfully in less than 5 min. Hosted by imgur.com
    Saturday, May 1, 2010 5:16 PM
  • Just in case someone stumbles upon this in the future...

     

    The problem was a 500GB Seagate Hard drive. The drive is fine, it works on any computer with the exception of Home Server and only when transferring to the pool. As you can see in the graphs above.

    Who knows why it is like that, but I can tell you for sure it was a MASSIVE pain to figure it out. Multiple SATA controllers, multiple motherboards, installs, blah blah blah.

    If you're having the same problem I was, may God have mercy on your soul... 

    • Marked as answer by inXorable Saturday, May 22, 2010 2:23 PM
    Saturday, May 22, 2010 2:22 PM