Home Server Backup corruption

All replies

• 

  

  0

  Sign in to vote

  S.M.A.R.T. only detects certain types of issues; overall a disk that's reported as unhealthy has a good chance of failing sometime soon, but a disk reported as healthy may also have problems. Google did some research a few years back and determined that S.M.A.R.T. isn't really a good indicator of long-term disk reliability.

  Please run chkdsk on all the disks in your server, then another backup repair and a cleanup.

  ---

  I'm not on the WHS team, I just post a lot. :)

  Friday, June 11, 2010 9:51 AM

  Moderator
• 

  

  0

  Sign in to vote

  Ok so Ive run the chkdsk, it  took quite a while (I think around 9 hours) for 6TB (4 disks).

  From what I could tell it didn't find any errors. The report under Application event viewer all say:

  Chkdsk was executed in read-only mode on a volume snapshot.
  For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

  4 times, but no additional reports.

  Also from the first one I have the output from the dospromt:

  CHKDSK is verifying files (stage 1 of 5)...
  495872 file records processed.
  File verification completed.
  21 large file records processed.
  0 bad file records processed.
  0 EA records processed.
  46155 reparse records processed.
  CHKDSK is verifying indexes (stage 2 of 5)...
  1161835 index entries processed.
  Index verification completed.
  5 unindexed files processed.
  CHKDSK is verifying security descriptors (stage 3 of 5)...
  495872 security descriptors processed.
  Security descriptor verification completed.
  5027 data files processed.
  CHKDSK is verifying Usn Journal...
  36231360 USN bytes processed.
  Usn Journal verification completed.
  CHKDSK is verifying file data (stage 4 of 5)...
  495856 files processed.
  File data verification completed.
  CHKDSK is verifying free space (stage 5 of 5)...
  151105706 free clusters processed.
  Free space verification is complete.
   955779142 KB total disk space.
   350681192 KB in 67494 files.
   48196 KB in 5028 indexes.
    0 KB in bad sectors.
   626930 KB in use by the system.
   65536 KB occupied by the log file.
   604422824 KB available on disk.
   4096 bytes in each allocation unit.
   238944785 total allocation units on disk.
   151105706 allocation units available on disk.

  Which all looks good.

  After this I did a backup repair, which completed without issues. All successful.

  However, then I tried to do a cleanup.
  After a few moments I get the error

  An error in the backup service is preventing the cleanup operation

  In my console it says:

  Backup Service not running

   (which auto corrects itself after a few moments but still)

  In the event log (Application) I see the following errors:

  Faulting application whsbackup.exe, version 6.0.2423.0, faulting module whsbackup.exe, version 6.0.2423.0, fault address 0x0001dcd1.
  For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

   

  Got Error: whs.criticalProcess.WHSBackup.exe Type:WHS_NOTIFICATION_ADD Head:Backup Service is not running. Desc:Backup Service is not running on server ENDOR. 
  For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

   

  Got Error: whs.criticalProcess.WHSBackup.exe Type:WHS_NOTIFICATION_REMOVE Head:Backup Service is not running. Desc:Backup Service is not running on server ENDOR. 
  For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

  I looked at:
  http://support.microsoft.com/kb/946339

  Did the update now (but automatic updates is on so)

  I have tried rebooting, none of my disks were disconnected, didn't recently remove any disks, all disks are marked as healthy, And I had just repaied the backup database ...

  Looking at the event log a bit closer I did see some strange messages (source Remote Notification):

  Got Error: qsm.05a1cf45-103b-4c27-8360-d37b38058922.Object Type:WHS_NOTIFICATION_ADD Head:Storage Status Desc:Volume with name Primary Volume has failed. Please check hard drives and try to run Repair to fix errors for disk WDC WD10EADS-00M2B0.

  Got Error: qsm.442e91c0-64b8-4b0d-a9fc-eaa686a00b38.Object Type:WHS_NOTIFICATION_ADD Head:Storage Status Desc:Volume with name DE Secondary Volume has failed. Please check hard drives and try to run Repair to fix errors for disk SAMSUNG HD154UI.

  Got Error: qsm.51d4ad7f-2084-4c1b-831b-566781f8e1ce.Object Type:WHS_NOTIFICATION_ADD Head:Storage Status Desc:Volume with name SYS has failed. Please check hard drives and try to run Repair to fix errors for disk WDC WD10EADS-00M2B0.

  Got Error: qsm.60703525-6b7f-4751-aa94-c584828a1e2a.Object Type:WHS_NOTIFICATION_ADD Head:Storage Status Desc:Volume with name DE Secondary Volume has failed. Please check hard drives and try to run Repair to fix errors for disk SAMSUNG HD154UI SCSI Disk Device.

  Got Error: qsm.940178df-a3b0-431b-8f7e-77f3ed12e245.Object Type:WHS_NOTIFICATION_ADD Head:Storage Status Desc:Volume with name DE Secondary Volume has failed. Please check hard drives and try to run Repair to fix errors for disk WDC WD20 EADS-32S2B0 WD-WCAVY1522434.

  Scrolling through the event log, these only started on 5th of june. And its on all my hard disks on boot time (normally I do hibernate my server though). But again, when I look into home server console it says all disks are present and ok. I think it's rather impossible that all my 4 hard disks have failed. They are all not connected the same way either. 2 are on the mainboard and the other 2 on a PCI Sata card, so that can't be a single point of failure either. And like I said, it's only a message at boot time in event log, otherwise everything seems in order.

  So I'm at a loss what to do now, so far it seems I've only made things worse :(

  I did try to copy/start some files/programs from my shares, those seem to work. On the other hand it's only a few files of a whole bunch so no way to be sure.

  I donno if its worth trying to remove the complete backup database and start over ? Or if that won't fix my issues ?

   

  Edit:

  Today in the event log ive got 2 messages saying the following:

  The device, \Device\Ide\IdePort2, did not respond within the timeout period.

  Ive read somewhere that this could be caused by a bad PSU ? Something to consider or not ?
  Also how can I figure out which hard disk this is exactly ? Because I think it's not just counting up from the first one ?

  Something to take into account, I had a samsung disk in bay 3 until 2 weeks ago, it started to randomly disconnect so I moved it to bay 5 until I could try a new sata cable. I've not seen any drives disconnect at random though since then so I thought that issue was fixed for the time being. So it's possible that it was the drive afterall and that that one is giving the errors (the one being at IdePort2). But it might have been on port 2 (since it was disk 3) 2 weeks ago but now it should be on port 4 then ? Maybe this is a coincidence but I thought it was worth mentioning ...

  Saturday, June 12, 2010 6:28 AM
• 

  

  0

  Sign in to vote

  If you will follow that link I posted previously, you will see that it includes a cmd file which will run chkdsk with the repair option on every drive in your server. Running chkdsk in read only mode doesn't fix anything, so please run the cmd file in that FAQ, then another repair and cleanup. If chkdsk reports that it corrected errors (the FAQ also tells you how to find those error reports), you will need to evaluate them to determine if they reflect a transient issue or a hardware problem.

  If that fails and chkdsk still doesn't report problems, the backup database may be corrupted in a way that repair can't correct. In that case, delete the backup database, then reboot your server. The location of the backup database is mentioned in the Home Computer Backup and Restore technical brief, which is available for download through the Windows Home Server section of the Microsoft web site.

  ---

  I'm not on the WHS team, I just post a lot. :)

  Saturday, June 12, 2010 12:57 PM

  Moderator
• 

  

  0

  Sign in to vote

  Hey Ken,

  I did execute your batch file, In the eventlog I find

  Chkdsk was executed in read-only mode on a volume snapshot.

  For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

  but then 3 times:

  Chkdsk was executed in read/write mode.

  For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

  No other reports were found in the application (or any other) event log.

  But what I'm currently most concerned of is:

  The device, \Device\Ide\IdePort2, did not respond within the timeout period.

  For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

  And also 1:

  The driver detected a controller error on \Device\Ide\IdePort2.

  For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

  The problem is, which one is on IdePort2 ... I have a tranquil with 5 bays, bay 1 to 4 were taken, but because of some cable connection issues a few weeks ago (or so I though, maybe that was my wake up call) I moved the disk from bay 3 to bay 5 and for 2 weeks didn't have any issues (before then the disk disconnected 2 or 3 times)

  So I'm really starting to think hard drive failure here :(
  If it's the disk I changed around previously (probably best bet) I changed the cable and connector by changing the bay, so that can't be it. They are still both on the same controller, but so is another disk and I've not seen any other reports other than IdePort2.
  If its another disk then I'm clueless on which one. But I have the build in connectors on the mainboard which hold 2 disks and then the pci card which holds the other 2 and the error is always about the same disk, so is it safe to say that it's not the controller but the disk (and maybe the cable/connector if it's another disk than the one I moved previously) ?

  So any idea how I can find out which one is on IdePort2 ? Thanks

  Saturday, June 12, 2010 1:38 PM
• 

  

  0

  Sign in to vote

  ...
  So any idea how I can find out which one is on IdePort2 ? Thanks

  Tranquil should be able to help you with that. Given this: "The driver detected a controller error on \Device\Ide\IdePort2." it may be a motherboard issue (your disk controllers are undoubtedly on the motherboard). Again, Tranquil is most likely to be able to help you.

  ---

  I'm not on the WHS team, I just post a lot. :)

  Saturday, June 12, 2010 1:49 PM

  Moderator
• 

  

  0

  Sign in to vote

  Yeah been trying to open a support ticket there all day, but their support site is down :(

  I guess best time spent now is to remove em one by one from the server and start copying files over to other disks ...

  Saturday, June 12, 2010 1:51 PM
• 

  

  0

  Sign in to vote

  Can I ask an additional quick question ?

  So I shut down my WHS to connect the disks to my pc (vista) and copy data off it. From what I heard this should be perfectly possible, however when I connect the disk to my pc it doesn't find it. When I open disk management it says :

  "you must initialize a disk before logical disk manager can access it" and lets me choose between MBR and GPT. But shouldn't I be able to just read a disk from WHS without having to do any of this stuff ? I don't want to change anything on the disk incase my WHS can be fixed and I just want to put this disk back in ... Tried it with 2 drives so far, same problem.

  These drives were purchased new and put in the home server who formatted them. The 2 drives I tried were also connected to different controllers (one to the mainboard, to other to a pci card) if that makes any sense. Maybe the question for MBR of GPT is normal, but I was expecting for it to just work out of the box.

  Thanks !

  Saturday, June 12, 2010 2:13 PM
• 

  

  0

  Sign in to vote

  On 6/12/2010 9:13 AM, Darthy1980 wrote:

  > Can I ask an additional quick question ?

  >

  > So I shut down my WHS to connect the disks to my pc (vista) and copy

  > data off it. From what I heard this should be perfectly possible,

  > however when I connect the disk to my pc it doesn't find it. When I open

  > disk management it says :

  >

  > "you must initialize a disk before logical disk manager can access it"

  > and lets me choose between MBR and GPT. But shouldn't I be able to just

  > read a disk from WHS without having to do any of this stuff ? I don't

  > want to change anything on the disk incase my WHS can be fixed and I

  > just want to put this disk back in ... Tried it with 2 drives so far,

  > same problem.

  >

  > These drives were purchased new and put in the home server who formatted

  > them. The 2 drives I tried were also connected to different controllers

  > (one to the mainboard, to other to a pci card) if that makes any sense.

  > Maybe the question for MBR of GPT is normal, but I was expecting for it

  > to just work out of the box.

  >

  > Thanks !

  >

   

  I'll take a stab at this, although you should probably listen to what

  Ken says (over anything that I say) as he's more of an expert on this

  than I am.

   

  You won't be able to just "read" the disks in Vista, because of how WHS

  "formats" them. Since WHS spans your folders across all of the disks,

  it doesn't format them in the same manner that a Windows Client (like

  Vista) does. So, Vista will think the disk is not formatted.

   

  In one of your earlier posts, you said that you thought you had a bad

  cable connecting the drive in bay 3 to the controller. Have you

  purchased a new cable? If so, then I would put the drive back in Bay 3

  and try the new cable. Again, I'm not an expert (I've only done one or

  two things with a multi-bay system), but my understanding is that the

  server expects the drive to be in Bay 3, so it's not going to recognize

  it in Bay 5 or anywhere else for that matter. I could be wrong about

  that though.

   

  If you remote into the Server, and go into Device Manager, you should be

  able to find out which one is on which port that way. Thre will be two

  places you can look. One is "Storage Controllers" (Right-click on each

  one, and open it's properties. It may be under location). And the other

  is Disk Drives (Right-click each and look at it's properties). Again I'm

  not entirely sure, since I'm doing this on a system with one controller

  and USB drives.

   

  Hope this helps you, and I'm sure Ken or someone else with more

  knowledge and experience will correct me where I'm wrong.

   

  Have a great day:)

  Patrick.

   

  --

  Smile... Someone out there cares deeply for you.

  Have you updated your OS and Antivirus today?

  http://update.microsoft.com

   

  ---

  Smile.. Someone out there cares deeply for you.

  Saturday, June 12, 2010 3:56 PM
• 

  

  0

  Sign in to vote

  Hey Patrick, thanks for posting.

  I'm definitely going to follow Ken's advice and contact Tranquil as soon as their support site is back up. I'm guessing I might have to RMA. But before that happens I'd love to be able to get as much data off the disks as possible. For which I thought that you could just connect the disks to your pc and read them, I've seen it advertised in many places. But is it possible then to mount them without making the disk unreadable for WHS ? It's important for me to be able to recover my data. I'm currently trying to copy with the disks connected to the home server but it's only going at 3MB/sec (even though it's set as DMA it's currently only going in PIO mode), this way its going to day days, maybe even weeks. Because I don't know which disk is giving me issues I can't really remove the faulty one (which would hopefully make the copy fast again).

  Ive purchases another cable so I could try that, however it does seem to be reading data fine from it (or should I say, not fine but at least smoething, recognises it and stuff)

  I've checked the location and you can see some stuff there, but nothing that maps to IdePort2 ... it's a different naming format.

  Thanks for the suggestion though, will probably try the other cable back in bay 3 ... I'm running out of other things to try :(

  Saturday, June 12, 2010 4:36 PM
• 

  

  0

  Sign in to vote

  I'll take a stab at this, although you should probably listen to what Ken says (over anything that I say) as he's more of an expert on this than I am.

   

  You won't be able to just "read" the disks in Vista, because of how WHS "formats" them. Since WHS spans your folders across all of the disks, it doesn't format them in the same manner that a Windows Client (like Vista) does. So, Vista will think the disk is not formatted.

  Not true.  WHS uses NTFS reparse points for storage of files on secondary drives.  Every hard drive should be readable by any client that can read NTFS.  All of this is assuming that Tranquil isn't using RAID (which they are not supposed to because it's not supported).

   

  In one of your earlier posts, you said that you thought you had a bad cable connecting the drive in bay 3 to the controller. Have you purchased a new cable? If so, then I would put the drive back in Bay 3 and try the new cable. Again, I'm not an expert (I've only done one or two things with a multi-bay system), but my understanding is that the server expects the drive to be in Bay 3, so it's not going to recognize it in Bay 5 or anywhere else for that matter. I could be wrong about that though.

  If you remote into the Server, and go into Device Manager, you should be able to find out which one is on which port that way. Thre will be two places you can look. One is "Storage Controllers" (Right-click on each one, and open it's properties. It may be under location). And the other is Disk Drives (Right-click each and look at it's properties). Again I'm not entirely sure, since I'm doing this on a system with one controller and USB drives.

  Hope this helps you, and I'm sure Ken or someone else with more knowledge and experience will correct me where I'm wrong.

  Have a great day:)

  Patrick.

  --

  Smile... Someone out there cares deeply for you.  Have you updated your OS and Antivirus today?  http://update.microsoft.com

  ---

  Smile.. Someone out there cares deeply for you.
  

  Saturday, June 12, 2010 8:13 PM

  Moderator
• 

  

  0

  Sign in to vote

  No it's definitely not raid. Just the standard way of WHS.

  So this is another problem then. I managed to start copying most off the server by switching one of the disks back to DMA instead of PIO (which I think got switched somewhere during the chkdsk, but not sure). So far looks like I'll have most of the stuff.

  The fact that I can't read 2 disks (of which 1 should be completely trouble free) on my client pc (I only tried the 2) which are on separate controllers probably means something else is going on ... I'll contact tranquil and see what they say.

  Is there anything I can do to be able to read the disks anyways without compromising the ability for WHS to read them ?

  Is it possible the problem of not being able to read them on a client pc occured because of the chkdsk i performed ?

  Sunday, June 13, 2010 1:02 AM
• 

  

  0

  Sign in to vote

  No it's definitely not raid. Just the standard way of WHS.

  So this is another problem then. I managed to start copying most off the server by switching one of the disks back to DMA instead of PIO (which I think got switched somewhere during the chkdsk, but not sure). So far looks like I'll have most of the stuff.

  The fact that I can't read 2 disks (of which 1 should be completely trouble free) on my client pc (I only tried the 2) which are on separate controllers probably means something else is going on ... I'll contact tranquil and see what they say.

  Is there anything I can do to be able to read the disks anyways without compromising the ability for WHS to read them ?

   

  You could try data recovery (either a professional service - which isn't cheap - or a DIY app).

  Is it possible the problem of not being able to read them on a client pc occured because of the chkdsk i performed ?

  The only I way I know of is if the drive was so far gone that it was about to completely fail anyway.
  

  Sunday, June 13, 2010 4:06 AM

  Moderator
• 

  

  0

  Sign in to vote

  Ok thanks, hopefully I'll be able to copy everything then just through the network.

  That would mean 2 of my disks (not including the one that was giving me trouble in the first place, so basically 3 disks) were failing at the same time from different controllers ... that seems very unlikely (but not impossible I guess). They are 2 samsungs and 1 WD, the samsungs are not from the same lot (I was careful about that when buying them).

  What would be a good a way of determining that (after I've got all my data copied) ? chkdsk, smart all don't give any errors ...

  Sunday, June 13, 2010 8:39 AM
• 

  

  0

  Sign in to vote

  Ok thanks, hopefully I'll be able to copy everything then just through the network.

  That would mean 2 of my disks (not including the one that was giving me trouble in the first place, so basically 3 disks) were failing at the same time from different controllers ... that seems very unlikely (but not impossible I guess). They are 2 samsungs and 1 WD, the samsungs are not from the same lot (I was careful about that when buying them).

  What would be a good a way of determining that (after I've got all my data copied) ? chkdsk, smart all don't give any errors ...

  I agree that 2 or 3 drives failing simultaneously is extremely unlikely (if it did happen to you, go buy some lottery tickets quick  :)  ).  In any event, chkdsk /r run on each drive individually (either by using Ken's script or by pulling one drive out, connecting it to a client, then running it) should be sufficient.

  At this point, I would suggest you wait until you hear back from Tranquil.

  Sunday, June 13, 2010 3:43 PM

  Moderator
• 

  

  0

  Sign in to vote

  Well, after 2 weeks tranquil finally replied:

   

  Hello Jorn,
  I must apologize for the late response but for the last week or so the entire technical team has been involved in the release of a new product, the iXL.
  I did manage to read your ticket a couple of days ago and I want to thank you for sending such a detailed description of the problems you are experiencing.
  We have heard of similar problems and all have one common denominator, the missing hard drive error. Most of the other problems you experienced where caused by the fail of one of your drives (not always the same one). We have succeeded in replicating the problem under extreme circumstances (5X2 TB hard drives, transferring huge amounts of info for long periods of time) and have limited the problem to high BUS usage between the Sil RAID card and the motherboard. We are not sure if the problem is hardware or software as we are still working on it.
  I am sorry that I don’t have a solution at the moment and all workarounds we have attempted involved using less BUS “speed” (limiting the data transfer speed with third party software).
  I will keep you posted regarding any solution or development.
  Thank you for your patience.

  If I'm reading that correctly it could take a long time before they come up with a solution. I hope not ... I've requested if they can swap the unit since it's still under waranty. So we'll see how that goes.

  Saturday, June 26, 2010 7:53 AM