Folder Duplication /DE / File Protection

All replies

• 

  

  0

  Sign in to vote

  I've been reading all the Vail posts since Vail was released (especially in the Vail Deal Breakers & Complaints post) and now I'm ready to take the plunge and play with this beta. 

  Please bear with me as I'm trying to be as brief as possible, but explaining my situation fully, and what I understand as an issue in using Vail.  And if I'm comprehending these situations correctly, then I won't waste my time with the beta and I'll probably need to stay with v1 or move to another platform for my needs.

  As with most people that have posted here, I'm pretty concerned about File Duplication/DE and some of the potential issues.  I'd like to make sure I understand the potential pitfalls so I can plan to avoid them. 

  I'm one of those video enthusiasts who have a home built server to accomodate my large video library - about 900 DVDs and BluRays - which takes up about 8TB of space.  I have 1 DVD drive and 4 BluRay drives hooked to my server to speed the rips to my computer as I learned from the school of hard knocks of what NOT to do so I don't crash my home server :-)  With anywhere from 1/2 to 1 hour for a BluRay rip and 15 minutes or so for a DVD rip, I'm very symphathic to those individuals concerned about File Duplication & DE as in my case it would mean about 225 hours (A MONTH) to rebuild my library in case of a hard drive incident.  Contrary to some posts, I WANT TO AVOID RE-RIPPING MY COLLECTION AT ALL COSTS!  I'm sure most video enthusiasts do as well.  I know there are companies out there who sell WHS as a solution for large video libraries and would be concerned with this also.

  However, I'm a little different than most users - I have 2 complete sets of backups for my video library and use Carbonite for my items that can't be replaced or would be costly to do so, iTunes, photos, etc., which gets me to the 30TB of storage I have and a total of 3-4 sets of my data (I've been through too many data losses NOT to be paraniod).  I use Folder Duplication for immediate restoral and I backup my WHS to my Server Backup drives which are housed internally (I have a 20 drive case-they're hot swap drives).  I did this because I learned in WHS v1 about the dreaded file conflicts that I occassionally received.  I could do a restore for those files quickly and efficiently  However, in reading these posts, I feel that even my paranoid state of having multiple backups, I may not be adequately protected.

  Now, to make sure I'm understanding everything I'm reading correctly about Vail (FD - Folder Duplication):

  With files larger than 1 gig and FD turned on, FD really does nothing for these files (1 gig or >) as it splits it across multiple drives.  So if one drive fails, then you lost that part of file rendering the entire file useless, correct? 

  No, that's wrong.  If you have Folder Duplication enabled on a share, all data in that share is protected from a single hard drive failure.  (Otherwise, what would be the point of FD?) 

  Using the above scenario and the fact that I backup my entire file system to Server Backup drives, when I restore files, it will need to restore the ENTIRE file, not just the part of the file that was lost on the one drive, which means, since 80%+ of a DVD or BluRay are files that are larger than 1 gig, then pretty much the entire video library will need to be restored if one of my hard drives crashes, correct?

  Only if the file is split across multiple drives.  However, I personally tried copying a nearly 2 GB file to Vail (with 2 drives in the pool), went through the simulate-a-failed-drive routine and yet the file survived (which means it was only on the one drive, not split across multiple drives). 

  In reading the posts, my understanding is that it is currently taking about 1 day for 1TB of data to be restored on Vail, correct?

  I'm not sure, I haven't timed it... 

  In WHS v1, you could always "temporarily remove a server backup drive" and remove it from the system to be stored offsite.  In my readings of the posts, am I understanding that the only way to remove a server backup drive from the system is if you're using an external drive and a hot swap drive designated as a "server backup drive" won't work in this scenario?

   I'm not sure what you mean. 

  If I'm correct on the issues above, how would you recommend that I "protect" my video library to quickly and efficiently restore it (my definition of quickly is a few hours - no more than a day) WITHOUT using RAID?

  One of the common misconceptions is "protect" vs. "restore".  Strictly speaking, they are not the same.  Protect involves redundancy (RAID, Folder Duplication, etc.).  Restore involves... well... restoring from a backup.  Ideally you want both.  If you are so concerned about time, you definitely want FD enabled for all shares (as there is no time elapsed at all when a drive fails, it just keeps on running).  Restoring 8 TB from backups will take a very long time no matter how you slice it.  (Not to mention you can only backup a share that is 2 TB or less, which, in your case, you couldn't do at all.) 

  I totally get that this is a "beta" and that some things aren't working correctly, not all features may be available, etc.  However, in my readings of the posts, these issues may be by design so I would like to clarify so that I have the correct facts, not someone's interpetation.

  I appreciate all of the comments from the Moderators - you guys "take a licking - but keep on ticking" in this forum.  You're much better people than I am - I couldn't do what you do - THANKS!

  P.S. Anyway to ensure these posts are working correctly?  In the Vail Deal Breakers/Complaint post you can't read them in chronological order - they're completely mixed up.  Makes it really difficult to read and review. 

  That post is in order for me.

   

  Wednesday, September 1, 2010 5:22 AM

  Moderator
• 

  

  0

  Sign in to vote

  The only supported way to ensure protection for your files from single drive failure, in either V1 or Vail, is to use duplication. Nothing has changed, because Microsoft has never guaranteed the contents of unduplicated shares in V1 if a drive fails. That you can see/access/copy the files on the individual drives rather than through the shares in V1 is an accident of design, one which some users have decided is an essential feature.

  For your bullet points:

  • If duplication is turned on for a share, you should not lose any files in that share if a single drive fails. However, any file which had a duplicated block on the failed drive is no longer protected, and is now vulnerable to loss if a second drive fails. It's impossible for users to determine what blocks of data are on what drive, but somewhere here I believe Microsoft has said that blocks are allocated from the drive with the most free space. Experimental evidence indicates this not to be true for small (a few GB) data sets, but it's what Microsoft says...
    In any case, you should assume that every file larger than 1 GB (or whatever the final block size is; I assume that's a tweak Microsoft could choose to make at some point) will be split across two or more drives. It's the pessimistic but intelligent approach, and taking that approach means you will rarely be disappointed.
  • Server Backup is completely different in Vail; it uses the Windows Server 2008 R2 Server Backup feature. It will back the file up, not the individual 1 GB blocks Drive Extender stores and deals with. And when you restore it, it will restore the file, not a single block. Backup/restore speeds will depend on your hardware and how much data there is to back up/restore. It's slower to back up your server the first time than in V1, but much faster for later backups (because Server Backup uses differencing vhds for backup). Restore from a backup will take a while. A restore of 1 TB of data could well take a day...
    If you're talking about just basic file copies, speeds will (again) vary depending on hardware and server workload, probably settling around 30 MB/s, or 9-10 hours per TB ignoring any validation of results.
  • You can detach server backup drives to take them off site in Vail.
  • If you have a multi-terabyte video library, there is no method a consumer can afford which will allow restoring it in the event of a disaster in a reasonable (by your definition) time. Consumers have unreasonable expectations in this area; pushing terabytes of data around can easily take days.

  If you have an extremely large volume of data RAID may be a better approach for you than duplication; hardware costs are lower with duplication for small amounts of data (say 4 or fewer drives, or around 3.5 TB of data in toto with 2 TB drives) but if you have 20 TB of data, you'll need at least 23 2 TB drives to hold it all. If you cut that down to 13 drives by using RAID, the cost of the extra 10 drives will more than pay for a really nice hardware accelerated RAID HBA with OCE, and a large external DAS case for the drives. (Note that your external array will almost certainly sound like a vacuum cleaner, and will frighten small children and pets, these devices are usually designed for use in a datacenter. But that's a price you pay for this kind of storage...)

  Regarding post order: posts in this forum are threaded, not displayed in chronological order. There is no way to change this.

  ---

  I'm not on the WHS team, I just post a lot. :)

  • Marked as answer by Paul Carvajal Thursday, September 2, 2010 10:01 PM

  Wednesday, September 1, 2010 12:43 PM

  Moderator
• 

  

  0

  Sign in to vote

  Regarding post order: posts in this forum are threaded, not displayed in chronological order. There is no way to change this.

  ---

  I'm not on the WHS team, I just post a lot. :)

  Each individual post within a thread is in chronological order for me (oldest on top, newest on bottom).
  

  Thursday, September 2, 2010 4:53 AM

  Moderator
• 

  

  0

  Sign in to vote

  And, if you use the NNTP Bridge from

   

  http://communitybridge.codeplex.com/ 

   

  or the Microsoft Bridge from

   

  https://connect.microsoft.com/MicrosoftForums/Downloads

   

  you can unselect "Group Messages by Conversation"

   

  in your newsreader (I use Outlook Express)

   

  to view all emails in chronological order in

   

  an unthreaded view.  But as stated,  by using

   

  the Forums, this is impossible.

   

  I use the Codeplex Bridge.

  
  --
   
  _______________

   

    BullDawg
    In God We Trust
  _______________

  <kariya21 [MVP]> wrote in message news:caf3bbd1-cd2c-459e-9928-bee15b4a0b02@communitybridge.codeplex.com...

  Regarding post order: posts in this forum are threaded, not displayed in chronological order. There is no way to change this.

  ---

  I'm not on the WHS team, I just post a lot. :)

  Each individual post within a thread is in chronological order for me (oldest on top, newest on bottom).
  

  ---

  BullDawg

  Thursday, September 2, 2010 5:51 AM
• 

  

  0

  Sign in to vote

  Bulldawg, it's quite possible that your newsreader offers additional threading views beyond what the forum software itself can support. that's not a feature of the forums, it's a feature of an external piece of software that not everybody uses (or can use; I don't have NNTP access most of the time). The NNTP bridge is, however, more of a transitional tool to help those who have historically used Microsoft newsgroups move over to the forums. And it's not without it's warts; if you use formatting in your newsreader the bridge and forum software will try to do "something intelligent" with it, usually reaulting in display difficulties down the line.

  In the forums, posts within a thread are ordered chronologically within each subthread. So if there are three replies to a post, those will be chronological. But if there are three sub-replies to the first of those, the three sub-repliess will be between the first and second reply, even if they were all posted after the third reply to the original post. This is intended to make clear who has replied to whom; unfortunately it breaks down when you exceed the nesting depth the forum display supports (around 6-7 levels, I think). The view I really want is a "header only" threaded view that preserves all levels of nesting (even if there are 30-40, as there may be in long threads). Then click on a header to expand the post.

  ---

  I'm not on the WHS team, I just post a lot. :)

  Thursday, September 2, 2010 12:42 PM

  Moderator
• 

  

  0

  Sign in to vote

  Bulldawg, it's quite possible that your newsreader offers additional threading views beyond what the forum software itself can support. that's not a feature of the forums, it's a feature of an external piece of software that not everybody uses (or can use; I don't have NNTP access most of the time). The NNTP bridge is, however, more of a transitional tool to help those who have historically used Microsoft newsgroups move over to the forums. And it's not without it's warts; if you use formatting in your newsreader the bridge and forum software will try to do "something intelligent" with it, usually reaulting in display difficulties down the line.
   
  In the forums, posts within a thread are ordered chronologically within each subthread. So if there are three replies to a post, those will be chronological. But if there are three sub-replies to the first of those, the three sub-repliess will be between the first and second reply, even if they were all posted after the third reply to the original post. This is intended to make clear who has replied to whom; unfortunately it breaks down when you exceed the nesting depth the forum display supports (around 6-7 levels, I think). The view I really want is a "header only" threaded view that preserves all levels of nesting (even if there are 30-40, as there may be in long threads). Then click on a header to expand the post.

  1. The organization of posts seen via the NNTP bridge is totally unrelated to that in the forum display.
   
  2. Although it could turn out to be the case, Microsoft has not presented the NNTP bridge as a transitional tool, but rather as a response to the outcry from many MVP's, and others, who do not like a forum interface, especially this one.
   
  3. In my experience, formatting disasters are much more common (and serious) in posts made directly in the forum interface than they are via the NNTP bridge.
   
  4. The threaded forum formatting that you describe obtains only if you have selected "threaded view" in your forum settings. Otherwise all the posts are in reverse chronological order.
   
  

  ---

  David Wilkinson | Visual C++ MVP

  Thursday, September 2, 2010 6:32 PM
• 

  

  0

  Sign in to vote

  David:

  1. Exactly.
  2. It was presented to me as a transitional tool. This seems obvious to me; Microsoft has been trying to eliminate their newsgroup support for years. Even if I loved newsgroups and hated online forums (I'm agnostic on the subject, other than that it's my participation in this forum that gained me MVP status) it would seem obvious. I would love an off-line interface of some sort, if Microsoft could only supply full functionality, but they've told me that this isn't possible in this version of the forum back-end. Without that I have only the forum software, and as I said previously, I don't have regular NNTP access anyway.
  3. The specific case I have in mind is quoting someone who's used some formatting and fonts and posted via the bridge. I've looked at the HTML that results, and it isn't pretty. And, of course, it's easy enough for someone who's dedicated to the task to produce arbitrarily bad results by editing the HTML view directly, or copying/pasting from Word to the forum editor, or (I suspect) doing the same with some newsreaders, so our points are probably equally valid. :)
  4. Good point. I set that 3 1/2 years ago and never looked back. :) Threaded is better than reverse chronological for me.

  ---

  I'm not on the WHS team, I just post a lot. :)

  Thursday, September 2, 2010 7:56 PM

  Moderator
• 

  

  0

  Sign in to vote

  The one thing I don't like, even in NNTP newsgroups, is the use of HTML in posts. It tears things up there just as much as you say it does in forums.

  ---

  Nancy Ward
  Windows 8 BetaFerret

  "Ken Warren [MVP]" wrote in message news:a60a5d00-8437-4d9b-9b34-dddd51bbcf11@communitybridge.codeplex.com...

  David:
  
  1. Exactly.
  2. It was presented to me    as a transitional tool. This seems obvious to me; Microsoft has been trying to eliminate their newsgroup support for years. Even if I loved newsgroups and hated online forums (I'm agnostic on the subject, other than that it's my participation in this    forum that gained me MVP status) it would seem obvious. I would love an off-line interface of some sort, if Microsoft could only supply full functionality, but they've told me that this isn't possible in this version of the forum back-end. Without that I have only the forum software, and as I said previously, I don't have regular NNTP access anyway.
  3. The specific case I have in mind is quoting someone who's used some formatting and fonts and posted via the bridge. I've looked at the HTML that results, and it isn't pretty. And, of course, it's easy enough for someone who's dedicated to the task to produce arbitrarily bad results by editing the HTML view directly, or copying/pasting from Word to the forum editor, or (I suspect) doing the same with some newsreaders, so our points are probably equally valid. :)
  4. Good point. I set that 3 1/2 years ago and never looked back. :) Threaded is better than reverse chronological for me.

  ---

  I'm not on the WHS team, I just post a lot. :)

  ---

  Nancy Ward

  Thursday, September 2, 2010 8:28 PM
• 

  

  0

  Sign in to vote

  Ken,

   

  I use Outlook Express that is part of WHSv1 through an Addin called Program Launcher.  The message store is on a non-storage pool drive and is backed up to an off site location daily through Hamchi VPN using Rich Copy also through the Program Launcher Addin.  The off site computer is also backed up nightly by WHS through the Hamchi VPN. This works well for me as I have a 100 Mbps up/down fiber connection at both ends here in Korea. I work at two different locations, so I maintain two households.  Sometimes, I do use the unsupported method of a Remote Desktop Connection instead of the Program Launcher Addin.

  
  --
   
  _______________

   

    BullDawg
    In God We Trust
  _______________

  <Ken Warren [MVP]> wrote in message news:0bb1907e-ef2b-45b3-b040-9812a11f783f@communitybridge.codeplex.com...

  Bulldawg, it's quite possible that your newsreader offers additional threading views beyond what the forum software itself can support. that's not a feature of the forums, it's a feature of an external piece of software that not everybody uses (or can use; I don't have NNTP access most of the time). The NNTP bridge is, however, more of a transitional tool to help those who have historically used Microsoft newsgroups move over to the forums. And it's not without it's warts; if you use formatting in your newsreader the bridge and forum software will try to do "something intelligent" with it, usually reaulting in display difficulties down the line.

  In the forums, posts within a thread are ordered chronologically within each subthread. So if there are three replies to a post, those will be chronological. But if there are three sub-replies to the first of those, the three sub-repliess will be between the first and second reply, even if they were all posted after the third reply to the original post. This is intended to make clear who has replied to whom; unfortunately it breaks down when you exceed the nesting depth the forum display supports (around 6-7 levels, I think). The view I really want is a "header only" threaded view that preserves all levels of nesting (even if there are 30-40, as there may be in long threads). Then click on a header to expand the post.

  ---

  I'm not on the WHS team, I just post a lot. :)

  ---

  BullDawg

  Thursday, September 2, 2010 10:11 PM