locked
A little concerned with WHS allocating storage RRS feed

  • Question

  •  

    I Installed an add-in (disk management) that showed me at what level my drives are being used.

     

    I have one internal drive and two external drives

     

    Size and usage

    System- internal : 232gb- 10% used

    External drive-1:      465gb- 21% used

    External drive 2:   186gb- 90% used

     

    I understand that WHS is to spread around the storage and the shadowing but it seems to be

    concentrating on the smallest drive first.

     

    Am I OK? what happens when it reaches 100%, not sure if that is good

     

    Can someone reasure me or is there something to worry about

     

    All drives are showing healthy

     

     

     

    thanks in advance

    Friday, January 25, 2008 2:23 AM

Answers

  • You're asking a different question now. Smile

    Microsoft recommends that you use your largest drive as your system drive at the time you install Windows Home Server. The thought is probably that you'll size your server for your current needs; if you have several PCs with lots of media, those needs might be quite high, so you might use two or three 500 GB drives to start.

    Microsoft doesn't say that you should change out your current system drive every time you add a new, larger drive. When that server with two or three 500 GB drives is approaching full, you're probably starting to record and copy HD media to your server, so you might add a 1 TB drive or two. But you just add that drive to the storage pool, you don't remove your system disk and replace it. WHS won't load balance across the new drive(s), per se (some things will move right away, some typically won't) but over time files will start moving to that drive from the other drives, to even out usage a bit.

    Should your system drive fail, by all means consider buying the largest drive currently available and using that as the new system drive. But there's no need to reinstall WHS every 6 months just because you're adding more storage.
    Friday, January 25, 2008 8:31 PM
    Moderator

All replies

  • It's likely that your smaller external drive was the first one connected, right? If so, that's where your backup database probably resides, and that's why it's mostly full.
    Friday, January 25, 2008 3:17 AM
    Moderator
  • As Ken says, you're fine. I should probably take out that % usage bar Wink

     

    Have a read of the Drive Extender whitepaper: http://download.microsoft.com/download/2/F/C/2FC09C20-587F-4F16-AA33-C6C4C75FB3DD/Windows_Home_Server_Drive_Extender.pdf

     

    Friday, January 25, 2008 5:38 AM
    Moderator
  • Okay so I've read the paper and understand that Microsoft doesn't see it as a problem, however if one accepts that there are "real" benefits to defraging a hard drive then I believe there is the potential for a problem. By design WHS uses the smallest disk first, and if one has a multi drive system with drives of very different size, it is easy and in my experience normal to have one drive at 90% usage or more and the other(s) at 30% ~ 40%.

    This creates problems for disk defragers. Diskeeper 8 will even suggest that one will benefit from having more free space, but if you happen to have a 1TB drive and a 250 GB drive WHS in my experience will load up the 250 GB drive first.

    Simply hiding this info from the user, doesn't seem to be the answer to me. Yes without knowing this, some users will have less to "worry" about. And some users might not notice that as they use their system it gets slower and slower, others will however. Even on a two drive system with the exact same size drives, WHS will load the pool drive, resulting in a system with the non system drive loaded to 90% and the system drive with less than 50% in my experience.

    Which makes me really question why we are told that the system drive should be the biggest, when by design WHS will load the D:\ drive only when it can no longer store to other pool drives.
    Friday, January 25, 2008 2:28 PM
  • The system drive should be the largest because it's used as the "landing zone" for files being copied to the server. Tombstones start out as the full size of the file being copied (which could be several GB, if we're talking about a ripped DVD, virtual machine disk, etc.), and during high-I/O periods, the actual file may be written to D: and migrated off later. Plus, Vista will look at the free space available before starting a file copy, and because of the way WHS deals with the storage pool, that's limited to the free space on D:.
    Friday, January 25, 2008 4:39 PM
    Moderator
  •  Ken Warren wrote:
    The system drive should be the largest because it's used as the "landing zone" for files being copied to the server. Tombstones start out as the full size of the file being copied (which could be several GB, if we're talking about a ripped DVD, virtual machine disk, etc.), and during high-I/O periods, the actual file may be written to D: and migrated off later. Plus, Vista will look at the free space available before starting a file copy, and because of the way WHS deals with the storage pool, that's limited to the free space on D:.


    I am not disputing anything you have said. However when it comes to applying this theory, I appear to see it different than you do. Let's take a user with a EX470 with a 500GB drive, who goes out and buys a second drive of say 1 TB. Now if the user follows the white paper suggestion he will pull the original drive, install the new drive and load the OS onto it, and then add the old drive back as a "pool" drive. WHS will then load all the files to the 500 GB "pool" drive. Leaving the 1 TB drive mostly empty. Most users are not going to be moving 100 GB or bigger files and the WHS just lacks the throughput, IMHO to move 100 plus GB files on a regular basis. I believe for most users buying a second 500 GB drive and a smaller say 200 GB drive for the system would make more sense. One would then have two pool drives, (giving about 1 TB in the pool) and their files I believe would be safer, while the 150 GB or so of free space on the system drive should appear as more than big enough for most normal users.

    It still doesn't address WHS tendency to load one of the two pool drives heavier than the other or the fragmentation this causes. As a media server, I believe most users will spend more time pulling files from a EX47x than sending files to a EX47x. As a backup device, only one computer at a time is backup, and again most computers that I know of have many small files and relatively few files of 10 GB or more. And still far fewer files of 100 GB plus size. Does one really need, a 1 TB disk to appear to have enough free space, to copy files to? Is the 1 TB disk being used in the most effective way when it is used as the primary disk in a EX470? I think not....
    Friday, January 25, 2008 6:18 PM
  • You're asking a different question now. Smile

    Microsoft recommends that you use your largest drive as your system drive at the time you install Windows Home Server. The thought is probably that you'll size your server for your current needs; if you have several PCs with lots of media, those needs might be quite high, so you might use two or three 500 GB drives to start.

    Microsoft doesn't say that you should change out your current system drive every time you add a new, larger drive. When that server with two or three 500 GB drives is approaching full, you're probably starting to record and copy HD media to your server, so you might add a 1 TB drive or two. But you just add that drive to the storage pool, you don't remove your system disk and replace it. WHS won't load balance across the new drive(s), per se (some things will move right away, some typically won't) but over time files will start moving to that drive from the other drives, to even out usage a bit.

    Should your system drive fail, by all means consider buying the largest drive currently available and using that as the new system drive. But there's no need to reinstall WHS every 6 months just because you're adding more storage.
    Friday, January 25, 2008 8:31 PM
    Moderator
  • Well now if one reads the white paper on the Windows_Home_Server_Drive_Extender, one can find, (on page 9/20)
     
    Note The primary data partition in a home server should be as large as possible for two reasons:
    1 You want to provide sufficient space to grow the file table for all of the files that you will store on your home server.
    2 Windows Vista® and other home computer operating systems check to see if there is adequate space on the primary data partition prior to starting a copy operation.

    Now techincally you are right, Microsoft doesn't say to exchange the drives, but I believe the above could be easly taken to imply that one should....

    And clearly you are stating that if you are going to reinstall the OS do it on the biggest drive, IMHO. But why? I believe that most of that 1 TB drive will not be used, untill all other drives are filled. That 1 TB drive will be the last drive used, in most cases. In a case where there is a 1TB drive and a 500 GB drive the 500 GB drive will be filled to 90 % capacity and a sigificant part of the free space will be locked and not usable for disk defragmentation. So the files simply get fragmented on the 500 GB disk, but it happens over time. Now using the biggest disk as the pool disk delays as long as possible this point in time, to my mind.
    Friday, January 25, 2008 9:18 PM
  • I quote myself:
     Ken Warren wrote:
    Should your system drive fail, by all means consider buying the largest drive currently available and using that as the new system drive.
    I proposed a specific circumstance in which I would replace a system drive of size X with a larger drive, not a general one. If my operating system is failing for some reason (probably because I installed too much stuff) I can perform a server reinstallation on the current hardware. Only if I have to physically replace my system drive would I go with a larger drive.

    Not that I disagree with the concept that there's a practical limit to the "system drive is the largest" requirement. I think that point is probably reached at whatever the storage price/GB "sweet spot" is, currently 500 GB. A year ago it would have been 320-400 GB.
    Friday, January 25, 2008 10:41 PM
    Moderator
  •  Sam Wood wrote:

    As Ken says, you're fine. I should probably take out that % usage bar

     

    Have a read of the Drive Extender whitepaper: http://download.microsoft.com/download/2/F/C/2FC09C20-587F-4F16-AA33-C6C4C75FB3DD/Windows_Home_Server_Drive_Extender.pdf

     



    First let me say I would like to know who writes this stuff.  I read the white paper.  The thing about hardware raid simply displays immense ignorance about how RAID actually works, unless the writer was referring to "Raid 0" which is not raid at all, in the same manner that JBOD is not RAID.

    http://en.wikipedia.org/wiki/RAID
    http://en.wikipedia.org/wiki/Standard_RAID_levels

    In the case of Raid 1 it is a simple "mirror" where exactly the same information is written to each disk, but it is done below the volume level.  If a drive fails, the alternate drive is used to supply the exact same information to the OS that is on the disk that failed.

    ANY raid 3 or greater writes data in stripes across the disks with "parity" information alternating in a round robbin fashion across the disks.  This means that if any single disk is lost, the parity information on the remaining disks absolutely, 100% ensures that the missing data can be recovered.  In this case the RAID software assembles pieces of physical disk surface and presents those pieces to the OS as a logical disk surface.

    To the OS a raid "volume" is exactly that, a LOGICAL formatted partition which presents to the OS a LOGICAL surface to be written to, divided into 4K or larger chunks.  The physical surface of the disk is NEVER presented to the OS at any time!!!

    It plainly states in the white paper that WHS writes to the volume, that the 4K chunk is the smallest storage location available etc.

    Yes, the replacement of a bad raid disk MAY BE slightly more difficult but only slightly.  In either case (RAID or WHS disk) the bad disk must be identified.  All raid systems have software dedicated to managing the RAID system and that software operates BELOW the volume level, not at the volume level.  I.e. if a raid has to be rebuilt, you replace the drive and the raid controller takes over and rebuilds the raid array PHYSICAL data storage.  It does the rebuild completely automatically, and it does NOT mess around with the LOGICAL volume per se, only with the MAPPING of logical to physical storage.  And I can tell you, without a doubt in my mind, that the rebuild of the raid array will take a miniscule fraction of the time that the "rebuild" of the newly inserted disk in WHS' methodology.  I have had disk failures in my Raid 6 arrays and the rebuild takes a few hours (for a 320g drive).  Rebalancing and rebuilding the shadow files in WHS could take days.

    You can think of RAID like a parking garage where you hand your keys to an attendant.  You have no clue where the car is parked, you just store your car in an available spot.  The attendant is the RAID controller, the parking lot is the raid array, your parking spot is on the volume, which DOES MAP to the parking garage, but you do not know, nor care how it is mapped.  What FLOOR (DISK) of the garage is your car on?  Who cares?  If a floor (DISK) of the garage has to be rebuilt the cars are just moved around but you don't know nor care where they are moved to.  In a perfect analogy COPIES of PIECES of your car would be parked on every single floor with extra copies of pieces of your car scattered around but we won't go there.  Read the link if you need to understand it better.

    Once you understand how RAID REALLY works, then some interesting options arise.  For example, file duplication can just go away.  The RAID controller is responsible for ensuring data integrety and does so MUCH more efficiently that a simple "duplication" strategy, although RAID 1 is in fact a simple duplication strategy, but even in that case (assuming a hardware controller) it does so MUCH (an order of magnitude) more efficiently.  If you have the luxury of defining a fully expanded RAID array (maximum possible number of drives already in place and formatted), then a single large volume can be used for WHS.  In that case there are NEVER any tombstones!  Tombstones only exist if a file is copied off to another logical volume and since the RAID array is presented to the OS as a SINGLE large volume, tombstones disappear (never created).  Balancing goes away.  "Smallest volume" goes away.  Calculations to discover which volume to place a file on goes away.

    Now, understand that if you then present additional volumes to WHS such as through a USB drive, you immediately start creating tombstones, balancing etc. But if you only ever present WHS with a single HUGE volume, then that entire issue should just never exist at all.  Files are written onto the main (and only) partition.  There is no place to copy them off to so they just sit there.

    At least that is my take on the thing.  It is really too bad that we don't get some of the design team talking to us on these forums.  They could attest to or dispute that interpretation but without insider knowledge one has to infer how it works and my inference says that RAID as a single huge volume managed by a hardware RAID controller is the fastest possible WHS server. 

    I KNOW how RAID works, and it ain't how the white paper would lead you to believe.  I KNOW that the OS does NOT poke down to the physical drive level on a RAID array.  The raid drivers simply do not allow the OS to do so!!! 

    I INFER that toombstones are never created in the presence of only one single volume.  Since there can be no copy, I INFER that all the related overhead just goes away.  Until a design team member comes along to dispute it, that is what I am left to believe.
    Saturday, January 26, 2008 3:05 AM
  • Hi John.

     

    I'm an infrastructure engineer by day, and as such I deal with RAID on a daily basis. Obviously, you have quite a bit of experience in that area as well.

     

    Even though I understand RAID intimately (and my desktop PC has a RAID1 data volume), I prefer the way WHS handles disks for bulk storage of media.

     

    Using DE and discrete volumes as mountpoints means I can slap whatever free disks I have lying around into the server without worrying about matching the existing drives or expanding a RAID volume. I can buy new storage a single disk at a time.

     

    With DE, I also get to choose which data to protect (at the share level at least), rather than an entire volume.

     

    Yes, DE has overhead. But so do cheap RAID controllers (and 'cheap' is nice way of describing the RAID controllers bundled with consumer motherboards). Expensive RAID controllers blow the pants off DE write speed, obviously, but we're talking about cheap, bulk, consumer-grade storage.

     

    I think we need to remember who this product is aimed at; it's not us.

    Saturday, January 26, 2008 3:17 AM
    Moderator
  •  Ken Warren wrote:
    It's likely that your smaller external drive was the first one connected, right? If so, that's where your backup database probably resides, and that's why it's mostly full.

     

     

     

    Actually the system drive was first it is internal, after completing the server install I add both the external drives about the same time.

     

    I was considering removing the smaller drive from the data base forcing WHS to relocate on to one of the larger drives then bringing it back on line as a reserve drive. I know it will take some time but willing to try it.

     

    what do you think, good idea?

     

    by the way I am at 94% full now on thes smaller drive.

    Saturday, January 26, 2008 2:49 PM
  •  xbsims wrote:

     

    I was considering removing the smaller drive from the data base forcing WHS to relocate on to one of the larger drives then bringing it back on line as a reserve drive. I know it will take some time but willing to try it.

     

    what do you think, good idea?

     

    by the way I am at 94% full now on thes smaller drive.

     

    It will take awhile, but that's essentially what the "Remove Drive" functionality is for. I've tested it a lot of different ways, and DE always moves the backup database (if it's present on the disk) to another disk.

    Saturday, January 26, 2008 7:35 PM
    Moderator
  •  Now I could be way off base here, but my assumption with the "Load Balancing" was that it was moving data between the drive pool drives to allocate rougly the same amount of data on each drive, where possible (of course keeping your redundant data requests in mind) -

    I have 8 hard drives in my drive pool currently. All Internal SATA (6 X 500GB & 2 X 1TB)  - since I am only storing about 1TB right now I have plenty of free space. I have 2 of my 500GB drives that are almost at 100%, and all of my other drives are sitting at almost 0% -

    Given the above configuartion I would much prefer my data be spread to all of the drives evenly when possible. In the scenario above if one of my 500GB drives that is actually being used does fail, I have essentailly lost half of my data - if it were spread evenly I would lose 10-15% of my data. Seems like the current "Load Balancing" places alot of "eggs" in only a couple of baskets.... 
    Dorkboy
    Monday, February 16, 2009 3:48 AM
  • Hi,
    data on Shared folders, for which you have enabled duplication, will not go lost due to a failed harddisk, unless both disks with the data and the duplicates fail at the same time (which may happen due to a simple overvoltage) or the disk fails in away that broken data gets still duplicated and you have no external backup.
    You should not add a bunch of drives at once, if you don't need the capacity, since they will sit there idle, consume power and are aging. The ideal starting disk configuration is one with 3 disks, if the backups and the data in shared folders fit on one disk.
    A spreading to all disks has benefits, if it comes to total losses, but makes it harder to perform manual recovery options, should those be necessary due to a broken OS with no Reinstall offered. I prefer nothing being lost than 20%.

    Best greetings from Germany
    Olaf
    Monday, February 16, 2009 8:08 AM
    Moderator