Fixing Database Inconsistency errors

Mon, 07 Jan 2008 02:15:44 Z

Fixing Database Inconsistency errors, Fix client backup database, Fix Backup service not running

NOT ALL ERRORS CAN BE FIXED USING THIS short manual. If you want to be prepared please use the method recommended in the Technical Brief for Backup and Restore pages 23 – 24

to save a copy of your database to an external disk on a weekly or monthly basis. If you have done this already you can skip A) and move directly to B).

If you want to enable Duplication for the Client Backup Database please read the instructions from Scott Hanselman in Windows Home Server Unsupported Feature - Backup Duplication .

If you have previously enabled duplication of the client backup database you can try to restore the Client Backup Database from a recent shadow copy.

PLease note: CLIENT BACKUP DATABASE DUPLICATION IS REQUIRED TO BE ABLE TO RESTORE THE CLIENT DATABASE FROM A RECENT SHADOW COPY. Also in some other cases restoring from shadow copies will not work due to issues with WHS shadow copy function. For background on this part please read FAQ: Previous Versions for files on Windows Home Server Shared Folders may not function correctly. NOTE: A method to REGAIN access to shadow copies that appear to be in accessible, as described in the FAQ, can be found overhere. This method will also work if the client backup database is NOT duplicated

Please note: This method (restoring from a shadow copy) will NOT work for HP Mediasmart since it has automatic shadow copies disabled. You can enable (WHS controlled) shadow copies on the HP Mediasmart by editing (or creating) a key:

Go to HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows Home Server\Storage Manager\Volumes\

Select the Volume which represents the D partition (select volume key, value for mountpoint should be D)

Edit or create the REG_DWORD key SnapPeriod Decimal value for this key should be 43200000.

Reboot to make effective. Check after 24h if snapshots are created(RDP into WHS, in explorer window right click drive letter, select shadow copies)

Restoring a previous version of the client Backup Database from a shadow copy (volume snapshot)

Start with saving a copy of the current set of database files, for instructions see "A" below.
If you skipped 1 you should first
- Connect to WHS by RDP or KVM
- Start, Run, type cmd, hit Enter
- In command shell type net stop pdl and hit Enter then type net stop whsbackup and hit Enter
On your WHS open explorer window
Browse to D:\folders
Right click the folder {00008086-058D-4C89-AB57-A7F909A47AB4}, select Sharing and Security
Select Share this folder, Apply, then OK
On the WHS desktop open the shared folders using the Shared Folders on Server shortcut
Right Click the folder {00008086-058D-4C89-AB57-A7F909A47AB4}, select properties
Now go to the previous version tab, and select the previous version you want, then click Restore.
Alternatively you can save a copy of the previous version to another location
After restore or copy has finished:
- Start, Run, type cmd, hit Enter
- In command shell type net start pdl and hit Enter then type net start whsbackup and hit Enter

Restoring the client backupdatabase if you can not restore it from a shadow copy.

A) Save the current set of database files

1) Connect to WHS by RDP or KVM

2) Ensure that Backup Cleanup is not running and that none of the home computers are being backed up, and do not access shares or copy data to the shares from client or remote (disable remote access temporarily)

3) Start, Run, type cmd, hit Enter

4) In command shell type net stop pdl and hit Enter then type net stop whsbackup and hit Enter

5) If you have a one disk system browse to D:\DE\folders\{00008086-058D-4C89-AB57-A7F909A47AB4}. For 2 or more disk system find the folder(s) C:\fs\X\DE\folders\{00008086-058D-4C89-AB57-A7F909A47AB4} where X represents drive letter or number assigned to mountpoint

6) Copy ALL files in this (these) folder(s) to a save location

7) NOTE1: the total size of all the files in this / these folder(s) should match the size of the folder D:\folders\{00008086-058D-4C89-AB57-A7F909A47AB4}.

NOTE2: The SIZE ON DISK of the folder D:\folders\{00008086-058D-4C89-AB57-A7F909A47AB4} divided by 4 should equal the number files of in this folder. If not demigrator is still busy moving files from this folder, or is unable to move some of the files from this folder. If nothing changes in 2 - 4 hours you may need to check C:\Documents and Settings\All Users\Application Data\Microsoft\Windows Home Server\logs\qsm.mmddyy.log and or q_de.log. Problem with these logs is that they are difficult to interpret; there's loads of "errors" which are not really problematic.

B) Identifying the cause of the problem.

1) Inspect the backup logs. You can find these in C:\Documents and Settings\All Users\Application Data\Microsoft\Windows Home Server\logs, named backup-mmddyy.log.

2) Look for file related errors. Some typical errors are:

1: Unexpected error 0x780 from CreateFile on D:\folders\{00008086-058D-4C89-AB57-A7F909A47AB4}\Commit.dat: The file can not be accessed by the system.

2: File Data.4096.0.dat does not exist, but is referenced by the following files: index.4096.dat

3: File Data.4096.10.dat_00796 should be named Data.4096.10.dat

C) Fixing the errors

1) Delete all files from the folder D:\folders\{00008086-058D-4C89-AB57-A7F909A47AB4}. This may result in an invalid file handle error for one or more files. If this happens:

i) Open command shell (see A2 above) and type fsutil reparsepoint delete filename and hit Enter (filename is the name of the offending file

ii) Try deleting again

iii) Repeat if necessary for other files

2) In command shell type net stop “drive extender migrator service”

3) COPY the files you have saved (A5 or previously saved backup set) to D:\folders\{00008086-058D-4C89-AB57-A7F909A47AB4}

4) Try restarting the backup service:

i) In command shell type net start PDL and hit Enter, then type net start whsbackup and hit Enter. Wait for a few minutes and then again type net start whsbackup. If it was still running you’ll receive a message it was already started. Otherwise it will start but probably fail quickly again.

5) If the backup service doesn’t fail within minutes after you started it there’s a good chance you fixed the database. Final check:

6) From the Windows Home Server Console, Settings, Backup menu run Backup Cleanup and watch the progress bar. It should gradually go to 100 % (please note this make take more then a few minutes)

i) If it finishes successfully look in the most recent backup log for any remaining errors

ii) If it stops running within seconds and the progress bar disappears please look in the most recent backup log for errors

7) If the backup service wasn’t started successfully or of there are any remaining errors:

i) In command shell type net stop PDL and hit Enter, then type net stop whsbackup and hit Enter

ii) For not existing file error or truncated / corrupt file error or file should be named error look for a file which has identical filename but has _nnnn (where n represents a number) added to the filename. If the original file is still there rename it, then rename filename_nnnn to filename. If there’s more then one _nnnn start with the largest one, if all are equal in size start with the most recent one. If this doesn’t fix it try the next copy

iii) Repeat for other files causing similar errors.

iv) Go back to C4

8) If the backup service started successfully and you can run backup cleanup without any errors please reboot WHS

9) If all errors are related to one specific client and you couldn’t fix your error you may try

i) Delete the machine from the WHS console (may require deleting all backups)

ii) Try running backup cleanup. After it finished (or failed):

iii) In command shell type net stop PDL and hit Enter, then type net stop whsbackup and hit Enter

iv) In explorer window go to D:\folders\{00008086-058D-4C89-AB57-A7F909A47AB4} and look for files referring to the machine you deleted, and delete these files manually if there are any left.

v) Reboot

Fixing Database Inconsistency errors

Mon, 07 Jan 2008 04:04:18 Z

Very nice, brubber!

However, please remember to use hyperlinks properly; just pasting a long link like you did messes up formatting for those with lower screen resolutions. I've fixed your link in this case.

Fixing Database Inconsistency errors

Mon, 07 Jan 2008 10:42:01 Z

Ken, thanks for polishing the hyperlink. I agree it looks better this way.

Fixing Database Inconsistency errors

Tue, 08 Jan 2008 00:31:11 Z

Questions:

1) The referenced Technical Brief instructions will require an external drive of 660GB minimum. This is the amount of data in the folder by doing a Properties on the folder. At the momemnt, this is impractical. Am I doing something wrong?

Part A - The backup of this folder is 438GB. Again, this is improactical at the momemnt. (I do not have enough offline storage for this.)

Part B - Is there any relation between this section and what needs to be done in Part C? The error that I am getting is:
Unexpected error 0x54e from OpenVolumeClusterFile on D:\folders\{00008086-058D-4C89-AB57-A7F909A47AB4}\{3B885706-80EC-4A89-8DDC-34A32ABAEFA5}.G.VolumeCluster.4096.dat: Unable to complete the requested operation because of either a catastrophic media failure or a data structure corruption on the disk.
Client Backup server failed at d:\qhsv1_rtm_qfe\qhs\src\backup\util\datafile.cpp(1035)

(By the way, both drives show up as healthy in the WHS console.)

Part C - The folder to be deleted contains 676GB. It seems that some data is going to be lost if the files from A are copied back to this folder. Why will this work without losing backup data.

Part C item 9 - I cannot delete the machine from the WHS console because the backup service will not start.

Thank you.

Fixing Database Inconsistency errors

Tue, 08 Jan 2008 01:32:12 Z

wscan wrote:

The referenced Technical Brief instructions will require an external drive of 660GB minimum. This is the amount of data in the folder by doing a Properties on the folder. At the momemnt, this is impractical. Am I doing something wrong?

Yes, it's impractical. However if you really need to recover you will have to invest in internal or external storage, could even be another networked system.

wscan wrote:

Part A - The backup of this folder is 438GB. Again, this is improactical at the momemnt. (I do not have enough offline storage for this.)

Please note I have recently updated A5 and A6. The backup database can be spread over more then one disk/ mountpoint (making failure of a single drive catatrophic for client backups) Total size of these folders should match or at least be close to the size of the folder D:\folders\{00008086-058D-4C89-AB57-A7F909A47AB4}

wscan wrote:

Part B - Is there any relation between this section and what needs to be done in Part C? The error that I am getting is: Unexpected error 0x54e from OpenVolumeClusterFile on D:\folders\{00008086-058D-4C89-AB57-A7F909A47AB4}\{3B885706-80EC-4A89-8DDC-34A32ABAEFA5}.G.VolumeCluster.4096.dat: Unable to complete the requested operation because of either a catastrophic media failure or a data structure corruption on the disk.
Client Backup server failed at d:\qhsv1_rtm_qfe\qhs\src\backup\util\datafile.cpp(1035)

Yes, if it can not open the file the reparsepoint may be messed up or pointing to the wrong file, or: If a database file get's corrupted the system sometimes creates another instance of the file (_nnnn). If the reparsepoint does not point the correct file this will also generate an error. By deleting the files and then copying back to D:\folders\{00008086-058D-4C89-AB57-A7F909A47AB4} the reparspoint will automatically be restored when demigrator starts moving backup files from the landing zone. Renaming _nnnn file with demigrator service stopped will effectively recreate correct reparsepoint.

wscan wrote:

Part C - The folder to be deleted contains 676GB. It seems that some data is going to be lost if the files from A are copied back to this folder. Why will this work without losing backup data.

Sizes should (more or less) match. Size on disk of folder D:\folders\{00008086-058D-4C89-AB57-A7F909A47AB4} divided by 4 should equal the number of files in this folder. If not demigrator service may have stopped or has not moved these files or cannot move the files. You may wish to check the appropriate log file

wscan wrote:

Part C item 9 - I cannot delete the machine from the WHS console because the backup service will not start.

You can after you moved the files to a save location and then deleted all backup files.

Fixing Database Inconsistency errors

Mon, 25 Feb 2008 11:19:05 Z

I have recently had to delete all backups from what I presume was a database inconsistency as WHS stopped backing up and knew nothing about any of the existing backups.

I do have an external drive with backup of WHS as instructed by the OP. I am a little recluctant to try the restore from this backup as it seems quite complex and am simply re-backing up all my (6) home computers. This will be completed in a few hours.

Given the complexity of the backup/restore process I was wondering if the upcoming PowerPack 1 will simplify backup and restore to and external source.

Fixing Database Inconsistency errors

Mon, 25 Feb 2008 13:25:46 Z

You will find complete details of Power Pack 1 on the Windows Home Server team blog.

Fixing Database Inconsistency errors

Mon, 25 Feb 2008 16:07:13 Z

Rach3 wrote:

I have recently had to delete all backups from what I presume was a database inconsistency as WHS stopped backing up and knew nothing about any of the existing backups.

Given the complexity of the backup/restore process I was wondering if the upcoming PowerPack 1 will simplify backup and restore to and external source.

If there are no major errors this is fairly simple process. In part C effectively all you need to do is delete one set of files and replace them with another set of files. Stopping the services is only to prevent the backup service starting to work on these files during copy.

The instruction is a bit lengthy because I have tried to take all possible scenario's into account. From what I know the upcoming Powerpack 1 will not really simplify this process.

I have recently added instructions for recovery based on the presence of volume shadow copies. This is also fairly simple to do.

Fixing Database Inconsistency errors

Mon, 25 Feb 2008 18:09:47 Z

brubber wrote:

From what I know the upcoming Powerpack 1 will not really simplify this process.

Please read the section titled "Home Computer Backups" in the blog posting I liked above. There will be a "database consistency fixer tool" which (from what little experimentation I've done, because I never experience the issue except on purpose) does everything that your guide above does.

(Which is not intended to belittle your guide. I suspect that PP1 includes this tool because of your guide. )

Fixing Database Inconsistency errors

Mon, 25 Feb 2008 21:58:20 Z

Ken Warren wrote:

brubber wrote:

From what I know the upcoming Powerpack 1 will not really simplify this process.

OK Ken, thanks very much for this info, I did read the blog post but simply missed that part! That's really a good thing.

Did you also notice the recent part I included about the use of shadow copies? It allowed me to restore a 5 day old working version the database. I'll post that one as a suggestion on connect too (later today or tomorrow). I think this method is to easy to put aside. All you need is timely warning. Perhaps it's even possible to do this automatically. Making use of the shadow copies actually adds a very cheap form of redundancy to the backup database.

As for the guide, I think it's best when there is no need for it at all.

Fixing Database Inconsistency errors

Tue, 12 Aug 2008 21:24:53 Z

It appears that the directory structure has changed, perhaps in PP1.

I have a two drive system and the 8086 folder is under

D:\DE\folders\{00008086-058D-4C89-AB57-A7F909A47AB4}

and under

C:\fs\4\DE\folders\{00008086-058D-4C89-AB57-A7F909A47AB4}

Does that mean that shadow copy is enabled?

Fixing Database Inconsistency errors

Wed, 13 Aug 2008 04:35:33 Z

sworcester wrote:

It appears that the directory structure has changed, perhaps in PP1.

I have a two drive system and the 8086 folder is under

D:\DE\folders\{00008086-058D-4C89-AB57-A7F909A47AB4}

and under

C:\fs\4\DE\folders\{00008086-058D-4C89-AB57-A7F909A47AB4}

Does that mean that shadow copy is enabled?

No, that just means your backup database is spread across both of those drives (and that is not new with PP1).