locked
Folder size limit of 2TB? RRS feed

  • Question

  • I've got a WHS with about 9.5TB of disk, 3.5TB of that free.  I was moving a 25GB file from one share to another and was getting an error returned of "not enough disk space"; didn't make sense with that much free space.  This operation was being done on the server, share to share.  Watching the operation run through a disk mgmt add-in, I could see the writes going to a disk that is 90% full (the rest are 50-60% full), and at 92%, the operation would fail.  I checked the destination folder and it totaled up to 2.0TB exactly, in the summary.  That got me thinking.  So I created a new shared folder and started moving the same file to that location instead.  Watching the disk, once again, the 90% guy was being written to.  The difference this time though was that it worked w/o issue. 

    So the question is, is there a limit to folder size?  I haven't tried yet, but I'd assume that if I split out the folder - it's a movies folder - into two folders for say, movies1 and movies2 - that'd resolve the issue for a while.  I can live w/that since I can aggregate them on the front end with WMC.

    Wednesday, November 10, 2010 4:42 AM

All replies

  • do you have folder duplication on?  a 2TB file in that folder would take 4TB.  .5TB more than you have.
    Thursday, November 11, 2010 4:29 AM
  • Sorry - I was probably clear as mud - The file in question was just 25GB.  The folder I'm copying to, the current total size of all files in that folder is 2TB.  When I created a new folder, the copy worked, it just won't allow me to copy to the folder with 2TB of files in it.  Both the intended destination folder and the test destination folder are on the same drive, so there is plenty of space.  And yes, I do have duplication on too.
    • Edited by elmo7 Thursday, November 11, 2010 5:16 PM
    Thursday, November 11, 2010 5:25 AM
  • Isn't 2TB at the outer limit of what 32bit addressing can handle?
    Thursday, November 11, 2010 8:30 AM
  • I can't answer as to why it is failing, but I can confirm that you can indeed have a folder over 2TB in size. On my backup whs, I have a single share folder of ~4TB.

    -timotl

    Thursday, November 11, 2010 4:59 PM
  • Ok, so I know at least that a folder share of > 2TB is possible.
    Thursday, November 11, 2010 5:18 PM
  • Partition Size Limitation

    The partition size is pretty straight forward.  On an MBR (Master Boot Record) disk, the locations where the partition sizes are stored are only 4 bytes long.  Since this is in hexadecimal, the largest value we can stuff in there is all F’s.  So the max value would 4,294,967,295 in decimal. 

    FF FF FF FFh = 4294967295d

    This maximum partition size is not in bytes, it is in number of sectors.  Since currently sectors are limited to 512 bytes, the maximum size ends up being 2 TB.

    4,294,967,295 sectors * 512 bytes/sectors = 2,199,023,255,040 bytes or 2TB.

    Number of Clusters

    The second limitation is harder to spot.  It is a limitation of NTFS.  NTFS is limited to (2^32 -1) clusters….no matter what.  The smallest cluster size possible is 512 bytes (1 sector).  So again the math leaves us at 2,199,023,255,040 or 2TB.

    (2^32)-1 = (4,294,967,296)-1 = 4,294,967,295 clusters

    4,294,967,295 clusters * 512 bytes/cluster = = 2,199,023,255,040 bytes or 2TB

    Maybe they came up with a way around this???

    I know WHS uses what amounts to a database to clump the drives into a D: drive that is larger than any one drive, but each individual drive added can still be no larger than 2TB.  This being the case, I don't know how a single folder cound be larger than 2TB, simply because of the addressing math involved.  I'm not saying it can't be done, I'm just saying I don't understand how.

    Please let me know if you figure this out.

    Monday, November 15, 2010 5:56 AM
  • Gary said:

    <stuff that's true but not relevant to Windows Home Server>

    Disk/partition table: Windows Home Server V1 uses MBR style partition tables, which are, as he notes, limited to 2 TB. However, those disks are "pooled" into server storage, so this limitation doesn't affect anything other than how much space Windows Home Server can use on a single disk, something that an end user will never see.

    Clusters: Windows Home Server formats disks with 4096 byte clusters. This allows for 16 TB in a single disk, however (again) this isn't a limitation for Windows Home Server, because of the way Drive Extender manages files. Files on D: are "tombstones", which are NTFS reparse points that contain information about where on the various disks in server storage the actual file (or files, if the file is in a duplicated share) lives. Files on other disks are files in a standard (NTFS) file system.

    As a result, a single folder can be arbitrarily large. I'm sure there a limit, and it's probably an unexpected number rather than something simple like 2 TB or 16 TB, as a result of using some particular datatype to refer to files, but I don't know what the limitation is, and it's certainly more than 2 TB, because there are many users with 10+ TB in a single share.

    N.B.: There is no "database" of files in the shares, unless you consider NTFS itself (with it's various metafiles) a database. Which is probably not too incorrect … :)


    I'm not on the WHS team, I just post a lot. :)
    Monday, November 15, 2010 1:51 PM
    Moderator
  • Elmo,

    I've individual shares in excess of 2TB.  I don't think what you are seeing is a limitation of WHS folder size...but WHS doesn't look at the free space on the disk before attempting to copy the file.  As the drive it selects is 90% full it probably doesn't have enough free space for a 25GB file, so fails with "not enough disk space" because WHS doesn't seem to look at the file size and other disks in the pool before determining which drive in the pool it should place the file on, and it doesn't split the file across disks in the pool (something which Vail will do).

    I've observed the same behaviour in my backup WHS - all my pool drives were >90% with the exception of the primary drive which WHS would not use whilst there was still space on the remaining drives.  Like you I had just under 3TB free spread mainly on the primary drive (1TB) and approx 10% free on each of the pool drives.

    As a temporary solution until I could get more drives online, I used DriveBalancer by Beefcake (wegotserved forums) to spread the free space on the primary drive against the pool so that all drives had a lower used %age, and could accommodate the 40GB files I was transferring.

    I don't think splitting the folder will make any difference to the problem as WHS would still try and put the large file on the next disk in the pool regardless of it's folder.

    Hope that helps.

    Monday, November 15, 2010 4:34 PM