locked
Robocopy from non-storage HDD to D:\shares on WHS PP2 - am I OK? RRS feed

  • Question

  • Hi,

    Just purchased WHS with PP1 and didn't get from reading the technical docs that writing to d:\shares was a no-no.  I had a home-grown media server using Windows XP so already had over 1TB of media files.  I quickly found during install I had to disconnect all but the 1.5TB drive I wanted WHS to format (it wanted to format all of them), then after install and updating from Windows Update, including Power Pack 2, I used robocopy (as I always did in the past) to copy all my media to the d:\shares folders.  I should mention that throughout I've had the server connected to a keyboard/mouse/monitor for direct access while setting it up.

    It seemed to work fine, and my media was accessible via UPnP devices like my D-Link Medialounge media bridge, so I used the WHS Console to add my two other HDD's (1TB and 500GB).  Almost immediately after the server started to freeze up every few seconds for about 20-30 seconds at a time and, though there was little CPU utilization (a few % from the demigrator) there was constant, heavy HDD activity.  I started to fear a virus or worm as I hadn't yet installed an A/V.

    I've been pouring thru these forums and tech docs and FAQs and didn't find the answer, except that I should NOT have copied to d:\shares but rather used the UNC \\server\shares - not sure how do even do that from internal HDD's using robocopy.  I also learned about the d:\de\shares folder and how the true datafiles (called "shadows") are supposed to be here while file pointers (called "tombstones") are in d:\shares - so I checked out d:\de\shares and found all my data there - as well as still in d:\shares, which I assume now are "tombstone" pointers.

    I finally removed the two HDD's, and soon after the heavy HDD utilization stopped and the server was usable again.  Re-adding the 1TB HDD didn't make the issue return.  Can I assume then that all the heavy activity was WHS copying the true datafiles from d:\shares to d:\de\shares and replacing them with tombstones?

    Finally, after I thought I figured it out and feel everything's OK, I tried downloading some add-in's to \\server\shares\software\add-ins but they wouldn't appear in my Console under Settings, Add-ins.  Investigating, I found they only existed in d:\shares\ and NOT in d:\de\shares.

    Being spooked by the whole affair, I removed the 1TB drive again and am now using robocopy to copy all my data from d:\de\shares folder back to my independent, "non-storage" HDD's.  I have another 1.5TB WD green drive ordered and on the way, so will wait for that before playing any further and definitely before turning on duplication.

    So my questions are:

    1) Is my system stable and OK as-is?  Am I saved by only having one HDD to start with?  Or would I best be served by reinstalling and copying using the UNC?
    2) How do I best copy data from internal, non-added HDD's to WHS-controlled HDD's?
    3) Why can't the Console find my add-in downloads when I do use the UNC?
    4) Is my data still all OK and am I doing the correct recovery procedure by copying themd from the d:\de\shares folder? (from the tech doc it looks like yes)
    5) Was my assumption correct that all the HDD activity and server freezes were the result of WHS trying to fix my incorrect filecopy process?

    Appreciate any feedback!
    Saturday, November 28, 2009 7:46 PM

Answers

  • The only supported way to copy data to your server is to copy to the network shares, rahter than the file system directly. So you could connect a drive to your server, then copy from it to \\localhost\<sharename>\etc. using whatever tool you like.

    As for copying into D:\DE\Shares\etc., that location is where Drive Extender puts the file shadow (or shadows on separate drives, if duplication is turned on for a share) for the tombstones that are in D:\Shares\etc. There is no supported way for an end user of Windows Home Server to even see that location (you have to log in to the server desktop, which isn't supported), so I suppose Microsoft didn't see a need to block all potential access routes.
    I'm not on the WHS team, I just post a lot. :)
    • Marked as answer by Bardmaster Saturday, December 12, 2009 7:14 PM
    Tuesday, December 8, 2009 2:53 PM
    Moderator

All replies

  • Hi,

    Just purchased WHS with PP1 and didn't get from reading the technical docs that writing to d:\shares was a no-no.  I had a home-grown media server using Windows XP so already had over 1TB of media files.  I quickly found during install I had to disconnect all but the 1.5TB drive I wanted WHS to format (it wanted to format all of them), then after install and updating from Windows Update, including Power Pack 2, I used robocopy (as I always did in the past) to copy all my media to the d:\shares folders.  I should mention that throughout I've had the server connected to a keyboard/mouse/monitor for direct access while setting it up.

    It seemed to work fine, and my media was accessible via UPnP devices like my D-Link Medialounge media bridge, so I used the WHS Console to add my two other HDD's (1TB and 500GB).  Almost immediately after the server started to freeze up every few seconds for about 20-30 seconds at a time and, though there was little CPU utilization (a few % from the demigrator) there was constant, heavy HDD activity.  I started to fear a virus or worm as I hadn't yet installed an A/V.

    I've been pouring thru these forums and tech docs and FAQs and didn't find the answer, except that I should NOT have copied to d:\shares but rather used the UNC \\server\shares - not sure how do even do that from internal HDD's using robocopy.

    You're not supposed to do that at all as logging into the server desktop at all is unsupported (for various reasons, one of which is what you're now finding out).

    I also learned about the d:\de\shares folder and how the true datafiles (called "shadows") are supposed to be here while file pointers (called "tombstones") are in d:\shares - so I checked out d:\de\shares and found all my data there - as well as still in d:\shares, which I assume now are "tombstone" pointers.

    I finally removed the two HDD's, and soon after the heavy HDD utilization stopped and the server was usable again.  Re-adding the 1TB HDD didn't make the issue return.  Can I assume then that all the heavy activity was WHS copying the true datafiles from d:\shares to d:\de\shares and replacing them with tombstones?

    It wouldn't copy to D:\DE\shares at all in that scenario.  If it was copying anywhere (which should only happen if your primary hard drive was nearly full), it would have been going to the secondary drive (which is not in that location).

    Finally, after I thought I figured it out and feel everything's OK, I tried downloading some add-in's to \\server\shares\software\add-ins but they wouldn't appear in my Console under Settings, Add-ins.  Investigating, I found they only existed in d:\shares\ and NOT in d:\de\shares.

    Being spooked by the whole affair, I removed the 1TB drive again and am now using robocopy to copy all my data from d:\de\shares folder back to my independent, "non-storage" HDD's.  I have another 1.5TB WD green drive ordered and on the way, so will wait for that before playing any further and definitely before turning on duplication.

    So my questions are:

    1) Is my system stable and OK as-is?

    It's hard to say.  Some users have had serious issues doing what you did while others seem to be ok.  (I can tell you from personal experience that when I was experimenting one day, I tried it as well.  It worked fine for my PCs, but my XBox 360 didn't have a clue what the file was until I re-added through the UNC path.)

    Am I saved by only having one HDD to start with?

    If you only had one drive the entire time, you would have a better chance of being ok.  But you added a drive at one point...

    Or would I best be served by reinstalling and copying using the UNC?

    If it was me, I would start over.

    2) How do I best copy data from internal, non-added HDD's to WHS-controlled HDD's?

    Through the UNC path, just like from anywhere else.  (That's what the shortcut on the server desktop is for.  :)  )

    3) Why can't the Console find my add-in downloads when I do use the UNC?

    Normally it should.  You might try resetting the Console.  If that doesn't work, try rebooting the server.

    4) Is my data still all OK and am I doing the correct recovery procedure by copying themd from the d:\de\shares folder? (from the tech doc it looks like yes)

    Technically, no (although it shouldn't hurt any either).  You should be copying from the UNC path.

    5) Was my assumption correct that all the HDD activity and server freezes were the result of WHS trying to fix my incorrect filecopy process?

    Since it was "working" ok before you added the drive, I would assume that either A) the added drive was bad or B) it was trying to free up space on your primary drive because it was nearly full.

    Appreciate any feedback!

    Saturday, November 28, 2009 8:07 PM
    Moderator
  • Thanks Kariya21! 

    I think I need more information on how Drive Extender and the DEmigrator works.  It does appear like it was able to recognize that the files in d:\shares were actual files and migated them to D:\DE\shares.  And the below exerpt from the DE technical brief almost suggests the DEmigrator needs to run for any new data transferred to the server, even if using the UNC path, so maybe not instantaneous? (as in my mystery of using the UNC path for Add-ins and the Console not registering the new msi files - after checking later both folders were synchronized)


    The Drive Extender Migrator service is run under the following conditions:

    ·         Every hour, or to be more specific – after resting an hour after the last completion.

    ·         When the Drive Extender Migrator service starts up – typically when you first turn on your home server.

    ·         When a shared folder is added or removed using the Windows Home Server Console’s Shared Folders tab.

    ·         When duplication on a Shared Folder is turned on or off.

    ·         When a hard drive is added to Server Storage using the Windows Home Server Console.


    I'm thinking I'm OK but will continue to run more tests.  Even though I added additional drives, they never had data migrated to them (which I know thanks to that great, free Disk Management Add-in), so all my files lived in D:\DE and on the main 1.5TB drive.  Like I said, I think I'll keep my files backed up separately for now (wish I'd never wiped 'em in the first place - lots of personal data like home videos and pics) and work with what I copied over "incorrectly" which seem to be fine and are recognized by my external devices (don't have an Xbox 360 - I have a D-Link DSM520 and a Samsung P3600, in addition to computers connected to my LAN).

    And I will consider reinstalling and recopying, but not until I learn more about the server and how DE works.  Got to go off on business for a week anyway, and the kids are watching Ben 10 right now off the WHS with no issues.

    Fascinating OS and so far very excited about its prospects!  Can't wait to try the Duplicator and PC Backup services.  Once my new HDD is delivered...

    Sunday, November 29, 2009 3:04 AM
  • OK so my new HDD came...bad sectors all over it - over 350GB!  Looks like the good folk at Newegg didn't treat their package like an egg.  It caused all kinds of problems when installed - couldn't get the Console to run, and somehow my admin profile got corrupted!  Still can't connect remotely using the Console!

    One RMA later (which I had to pay for the shipping...now is that right??) I'm waiting for hopefully a carefully-handled drive to try the experiment again.  I'm reinstalling WHS, to be sure.

    But I'd STILL like somebody else's opinion or advice on using Robocopy or some other fast, effective, and stable file-copying solution to copy from internal SATA to SATA.  There MUST be something better than a simple Windows Explorer copy-and-paste from the /shares folders!!

    Why can't I robocopy directly from my non-storage hard drives to D:\DE\SHARES?

    (ps my remote access is working like a charm since I hardcoded the port forwarding in my D-Link DIR-655 router and also hardcoded my server's ip address!)
    Tuesday, December 8, 2009 5:28 AM
  • The only supported way to copy data to your server is to copy to the network shares, rahter than the file system directly. So you could connect a drive to your server, then copy from it to \\localhost\<sharename>\etc. using whatever tool you like.

    As for copying into D:\DE\Shares\etc., that location is where Drive Extender puts the file shadow (or shadows on separate drives, if duplication is turned on for a share) for the tombstones that are in D:\Shares\etc. There is no supported way for an end user of Windows Home Server to even see that location (you have to log in to the server desktop, which isn't supported), so I suppose Microsoft didn't see a need to block all potential access routes.
    I'm not on the WHS team, I just post a lot. :)
    • Marked as answer by Bardmaster Saturday, December 12, 2009 7:14 PM
    Tuesday, December 8, 2009 2:53 PM
    Moderator
  • OK thanks Ken - though I must mention that logging in via RDP, which is supported, does allow a user (logging in as an admin of course) to see D:\DE\Shares.  Regardless, I understand we need to use the UNC. 

    I did a reinstall of WHS, which is actually quite slick (since it does a full reformat and fresh install of only the system partition and you don't need to worry about it wiping out your data partition), and everything is now working great!  There was no need to even consider recopying all my files because WHS saw the existing structure and synched/indexed everything cleanly.  Now I have no issues with copying to \\servername\software\add-ins and not actually seeing the Add-in in the Console right away, there are no startup errors, it's all good.

    Thanks to all!

    Brad

    Saturday, December 12, 2009 7:14 PM
  • ...
    logging in via RDP, which is supported
    ...

    Nope, not supported. Often recommended, and a very useful technique if you're careful, but it's not supported. Think of it as a functional Windows Home Server vermiform appendix. :)

    I'm not on the WHS team, I just post a lot. :)
    Sunday, December 13, 2009 4:58 AM
    Moderator
  • Can anyone please provide a more in-depth explanation to why "D:\Shares" is off limits, rather than saying that you're not supposed to do that - but use the actual share?

     

    Saturday, July 31, 2010 11:55 AM
  • Because D:\Shares is managed by Drive Extender, to allow the disk spanning and duplication functionality to work. Direct access to that location through a local file system (D:\shares\etc. rather than \\Server\etc.) allows software to use API calls for file manipulation which DE might not intercept, resulting in data being stored directly on the system drive.
    I'm not on the WHS team, I just post a lot. :)
    Saturday, July 31, 2010 1:13 PM
    Moderator
  • Hi Ken,

    Thanks for the reply. That is in fact what I've experienced. Though I think I've read somewhere that WHS as standard fills the disks, one after another. disk 1 full -> fill disk 2.

     

     

    Saturday, July 31, 2010 2:03 PM
  • Hi Ken,

    Thanks for the reply. That is in fact what I've experienced. Though I think I've read somewhere that WHS as standard fills the disks, one after another. disk 1 full -> fill disk 2.

    Yes, that's correct.  But that has nothing to do with whether DE is aware of the files or not.
    Saturday, July 31, 2010 2:47 PM
    Moderator