locked
MSS WHS Backup Database Recovery RRS feed

  • Question

  • Hello, I have a a backup database consistency error and need help.

    So here's the story, my HP mediasmart server's primary hard drive began failing. I had no choice but to do a server recovery with a new system drive ( http://www.mediasmarthome.com/article/10014/How-To-Restore-the-HP-MediaSmart-Server/ ). Before the recovery I imaged the dying hard drive to get a copy of the sectors that were still readable. Most of the damage was at the beginning of the disk.

    As a side note, after many failed attempts at server recovery and nearly giving up, I discovered that the procedure doesn't work with a USB drive plugged in. It would have been nice if they had mentioned that somewhere.

    After I got the server recovery to go it went without a hitch, the server rebooted and right now I see my old shares still there and my files appear to be there too.

    Unfortunately the backup database will not go. I know for sure that none of the back up database files were on the failing drive (I have an image). Here's an excerpt from the backup-xxx.log file after the recovery:

    Log

    [7/12/2008 12:58:54 PM ab8] Checker: Error: File Control.4096.dat does not exist, but is referenced by the following files:
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.1.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.44.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.45.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.46.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.47.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.48.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.49.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.5.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.50.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.51.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.52.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.53.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.54.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.55.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.56.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.57.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.58.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.59.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.6.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.60.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.61.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.62.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.63.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.64.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.65.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.66.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.67.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.68.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.69.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.7.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.70.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.71.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.72.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.73.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.74.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.75.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.76.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.77.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.78.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.79.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.8.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.80.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.81.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.9.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.14.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.15.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.16.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.17.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.18.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.19.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.2.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.20.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.21.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.22.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.23.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.24.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.25.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: GATEWAY.C.VolumeCluster.4096.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: GlobalCluster.4096.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: GlobalClusterLatest.4096.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: LAPTOP.C.VolumeCluster.4096.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: LAPTOP.E.VolumeCluster.4096.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: LAPTOP.F.VolumeCluster.4096.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: MAINDESKTOP-1.C.VolumeCluster.4096.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: MAINDESKTOP-1.D.VolumeCluster.4096.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: MAINDESKTOP.C.VolumeCluster.4096.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: MAINDESKTOP.E.VolumeCluster.4096.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.10.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.11.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.12.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.13.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.26.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.27.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.28.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.29.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.3.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.30.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.31.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.32.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.33.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.34.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.35.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.36.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.37.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.38.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.39.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.4.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.40.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.41.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.42.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.43.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Finished checking backup database consistency, 1 error(s) found.



    After comparing the current backup folder contents with the original pre-server recovery copy, I see 2 files missing:

    Missing Files from Backup DB

    Control.4096.dat      4,096 b

    Data.4096.0.dat       4,296,967,296 b


    I would of course like to recover my backup database without loosing too much data as it holds some important information. And luckily it looks like not too much actual data was lost. Only about 4 GB and judging from the file name it was probably a very early backup.

    What I need to know is the following:

    1. What is the Control.4096.dat responsible for ? It looks like a small file and is obviously being referenced from the Data.xxx.xx.dat files. This tells me that it might be possible to rebuild this file. Is it possible to get the format of this file or some sort of rebuilding process? For example, I have been in the past been able to recover NTFS partitions by parsing the RAW bytes of the MFT, is something similar possible here?

    2. Given that the data in Data.4096.0 is gone, is it possible to rebuild the backup database excluding any backups made that were referencing that file?

    3. What other tools or options are available for WHS backup database recovery that would fit this scenario? I know about the unsupported DB duplications features and the shadow copies that accompany it, I believe this doesn't apply here because I did not have those on.

    Thank you very much for any help offered, I really appreciate it.
    Saturday, July 12, 2008 6:52 PM

Answers

  • So, to put an end to this. Given my efforts above, armed with a hex editor, I was able to reconstruct the missing files and get my corrupt backup database to open and I am now able to pull files off.

    It would be a shame for the information discussed above to get lost so I'll probably be posting it somewhere else more permanent and also the promised 4th part.
    Monday, July 14, 2008 11:08 PM

All replies

  • AlexPi,

    My first thoughts if you have ANY data missing from a backup database, is that you are out of luck trying to rebuild.

    WHS uses Single Instance Storage, which in effect means that any file which appears anywhere more than once, is only stored in the backup once and all the other copies of that same file are replaced with a 'pointer' to that one stored file.

    Now, the first backups done, are the most important as they contain the first ever copy of any particular file. All other backups from other clients and subsequent backups, will only ever contain the pointers.

    If you look at one of the recent backups through your Console, it appears that it is a complete backup, however in reality, it just contains the latest files and pointers to all the previously backed up data.

     

    The Control.dat file contains info on 

     

      <Field Type="int" Name="BytesPerCluster" />
      <Field Type="int" Name="NextIndex" />
      <Field Type="int" Name="NextDataOffset" />

    amongst other items.

     

    Also, 4096.0.dat is usually that first, most important backup. However, if there were a large amount of data to backup, it would have 'overflowed' into the next 4096 file. (WHS saves backups in 4GB chunks).

     

    Others may well have more info which will help, but that's one of the reasons I copy off my backups to another location.

     

    Colin

    Saturday, July 12, 2008 7:15 PM
  •  AlexPi wrote:
    Hello, I have a a backup database consistency error and need help.

    So here's the story, my HP mediasmart server's primary hard drive began failing. I had no choice but to do a server recovery with a new system drive ( http://www.mediasmarthome.com/article/10014/How-To-Restore-the-HP-MediaSmart-Server/ ). Before the recovery I imaged the dying hard drive to get a copy of the sectors that were still readable. Most of the damage was at the beginning of the disk.

    As a side note, after many failed attempts at server recovery and nearly giving up, I discovered that the procedure doesn't work with a USB drive plugged in. It would have been nice if they had mentioned that somewhere.

    After I got the server recovery to go it went without a hitch, the server rebooted and right now I see my old shares still there and my files appear to be there too.

    Unfortunately the backup database will not go. I know for sure that none of the back up database files were on the failing drive (I have an image). Here's an excerpt from the backup-xxx.log file after the recovery:

    Log

    [7/12/2008 12:58:54 PM ab8] Checker: Error: File Control.4096.dat does not exist, but is referenced by the following files:
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.1.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.44.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.45.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.46.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.47.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.48.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.49.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.5.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.50.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.51.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.52.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.53.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.54.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.55.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.56.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.57.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.58.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.59.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.6.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.60.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.61.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.62.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.63.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.64.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.65.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.66.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.67.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.68.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.69.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.7.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.70.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.71.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.72.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.73.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.74.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.75.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.76.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.77.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.78.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.79.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.8.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.80.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.81.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.9.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.14.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.15.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.16.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.17.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.18.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.19.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.2.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.20.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.21.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.22.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.23.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.24.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.25.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: GATEWAY.C.VolumeCluster.4096.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: GlobalCluster.4096.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: GlobalClusterLatest.4096.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: LAPTOP.C.VolumeCluster.4096.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: LAPTOP.E.VolumeCluster.4096.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: LAPTOP.F.VolumeCluster.4096.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: MAINDESKTOP-1.C.VolumeCluster.4096.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: MAINDESKTOP-1.D.VolumeCluster.4096.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: MAINDESKTOP.C.VolumeCluster.4096.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: MAINDESKTOP.E.VolumeCluster.4096.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.10.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.11.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.12.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.13.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.26.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.27.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.28.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.29.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.3.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.30.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.31.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.32.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.33.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.34.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.35.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.36.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.37.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.38.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.39.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.4.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.40.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.41.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.42.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Error: Data.4096.43.dat
    [7/12/2008 12:58:54 PM ab8] Checker: Finished checking backup database consistency, 1 error(s) found.



    After comparing the current backup folder contents with the original pre-server recovery copy, I see 2 files missing:

    Missing Files from Backup DB

    Control.4096.dat      4,096 b

    Data.4096.0.dat       4,296,967,296 b


    I would of course like to recover my backup database without loosing too much data as it holds some important information. And luckily it looks like not too much actual data was lost. Only about 4 GB and judging from the file name it was probably a very early backup.

     

    You shouldn't need to restore your backups.  Just run the backup tool from your client PC to create a new backup database.

     

     AlexPi wrote:
    What I need to know is the following:

    1. What is the Control.4096.dat responsible for ? It looks like a small file and is obviously being referenced from the Data.xxx.xx.dat files. This tells me that it might be possible to rebuild this file. Is it possible to get the format of this file or some sort of rebuilding process? For example, I have been in the past been able to recover NTFS partitions by parsing the RAW bytes of the MFT, is something similar possible here?

     

    I don't like your chances of recovering your data without that data file there.  However, I would start with brubber's post Fixing Database Inconsistency Errors.

     

     AlexPi wrote:
    2. Given that the data in Data.4096.0 is gone, is it possible to rebuild the backup database excluding any backups made that were referencing that file?

     

    No.

     

     AlexPi wrote:
    3. What other tools or options are available for WHS backup database recovery that would fit this scenario? I know about the unsupported DB duplications features and the shadow copies that accompany it, I believe this doesn't apply here because I did not have those on.

    Thank you very much for any help offered, I really appreciate it.

    Saturday, July 12, 2008 7:23 PM
    Moderator
  • If I understand correctly, you have a complete copy of your backup database from before the errors and drive replacement. Is that correct? If so, you can probably restore it using the restoration portion of the database backup/restore method laid out in the Home Computer Backup technical brief. It that's not correct, then there may be another option, so let us know.
    Sunday, July 13, 2008 3:21 PM
    Moderator
  •  Ken Warren wrote:
    If I understand correctly, you have a complete copy of your backup database from before the errors and drive replacement. Is that correct? If so, you can probably restore it using the restoration portion of the database backup/restore method laid out in the Home Computer Backup technical brief. It that's not correct, then there may be another option, so let us know.


    No this is not correct. Unfortunately I have no backup pre-file loss.
    Sunday, July 13, 2008 10:19 PM
  •  ColinWH wrote:

    AlexPi,

    My first thoughts if you have ANY data missing from a backup database, is that you are out of luck trying to rebuild.

    WHS uses Single Instance Storage, which in effect means that any file which appears anywhere more than once, is only stored in the backup once and all the other copies of that same file are replaced with a 'pointer' to that one stored file.

    Now, the first backups done, are the most important as they contain the first ever copy of any particular file. All other backups from other clients and subsequent backups, will only ever contain the pointers.

    If you look at one of the recent backups through your Console, it appears that it is a complete backup, however in reality, it just contains the latest files and pointers to all the previously backed up data.

     

    The Control.dat file contains info on 

     

      <Field Type="int" Name="BytesPerCluster" />
      <Field Type="int" Name="NextIndex" />
      <Field Type="int" Name="NextDataOffset" />

    amongst other items.

     

    Also, 4096.0.dat is usually that first, most important backup. However, if there were a large amount of data to backup, it would have 'overflowed' into the next 4096 file. (WHS saves backups in 4GB chunks).

     

    Others may well have more info which will help, but that's one of the reasons I copy off my backups to another location.

     

    Colin



    I see your point about the first backup being the most crucial, but I hold out some hope that I can still recover some of the data even though that crucial first file is missing. My first backups are pretty old when I was still running XP and now I'm on Vista so perhaps the newer backups have no clusters from that first backup.
    Sunday, July 13, 2008 10:22 PM
  • I have been making good progress deciphering the WHS backup DB format.

     

    Each file seems to be divided into at least 4 parts:

     

    1. Header
    2. Values
    3. Definition
    4. Records

     

    Let's start with the Header

     

    Header

    Here's the header of a typical Control.4096.dat in Hex:

     

    08 00 00 00 08 00 00 00 04 00 00 00 00 02 00 00
    00 04 00 00 01 00 00 00 01 00 00 00 00 10 00 00
    00 10 00 00

     

    These seem to be little-endian 4 byte integers.

     

    Let's look at these 1 at a time:

     

    1. 08 00 00 00 - (8) Unknwon. Number of entries in the Header?

     

    2. 08 00 00 00 - (8) Unknwon. Sectors per cluster (512 bytes per sector * 8 sectors per cluster = 4096 bytes per cluster)?

     

    3. 04 00 00 00 - (4) File type. 4 = Control file.

     

    4. 00 02 00 00 - (512) Offset into Values section.

     

    5. 00 04 00 00 - (1024) Offset into Definition section.

     

    6. 01 00 00 00 - (1) Unknown.

     

    7. 01 00 00 00 - (1) Unknown.

     

    8. 00 10 00 00 - (4096) Unknown. Seems to be 00 00 04 00 (262144) for files > 64KB.

     

    9. 00 10 00 00 - (4096) Offset into Records section.

     

    The important ones here are 4, 5, 9. These let us parse the file further.

     

    Sunday, July 13, 2008 10:59 PM
  • Next let's skip to Definition.

    From the Header we can see where the Definition begins (in byte offset from beginning of file)

     

    Definition

    The definition is the most readable section here and is actually an XML text fragment.

     

    Let's look at the XML from Control.4096.dat:
     
    <File Type="Control">
     <Header>
      <Field Type="int" Name="BytesPerCluster" />
      <Field Type="int" Name="NextIndex" />
      <Field Type="int" Name="NextDataOffset" />
     </Header>
    </File>

     

    The definition ends with byte 0.

     

    This clearly is defining the format of the file. In this case, we can see the file type and the header fields. Each entry under the header tag is a specific reference to a field VALUE stored in the Values section. The type defines how we deserialize (or decipher) the value for that field.

     

    Now that we have our definition, we can start looking at the actual field values in the Values section.

     

     

    Sunday, July 13, 2008 11:10 PM
  •  

    Values (or field values) are kind of tricky in that they're serialized into binary so let's look at those
    now.

    Values

    The data in the Values section can be obtained by going to the offset specified in the header.

     

    Back to our infamous Control.4096.dat.

     

    The Values section in HEX:

     

    80 20 80 98 81 0b 80 c0 d7 a5 80 80 80 0b

     

    You'll understand why I chose the number of bytes that I did after understanding how deserialization works. The field definitions would lead you to believe that these are some sort of fixed integers (like in the header).
    They are not! In fact, they are specifically encoded integers, that are VARIABLE length.

     

    Let's look at the first field. It's type is int and it's name is BytesPerCluster. After doing a bit of Googling I figured out that there should be 4096 bytes per cluster. So how do you get 4096 out of Hex 80, or is it 80 20,
    or maybe 20 80 ?

     

    Let me spare you the gory details of how I figured this out, here's the secret:

     

    Each byte's first bit determines whether the next byte is part of the same value.

     

    So let's look at this in binary:

     

    80 = 10000000
         ^
         This means to read the next byte, we're not done.

     

    20 = 00100000
         ^
         This means we're done and the next byte is part of the next value.

     

    Also, because the first bit plays this special role, it does not compute into the actual value.

     

    Now that we know this, how do we calculate the value of these?

     

    The value is actually stored as little-endian whole numbers (not decimal floats), and the first bit does not play a role in the value calculation.

     

    So here's a simple deserialization algorithm:

     

    Step 1. Find the number of bytes participating in this value. So based on the information above, the number of bytes in this case is 2. Since the first encountered byte with its first bit set to 0 is byte number 2.

     

    Step 2. Go through each bit starting from the most-significant bit to the least significant bit adding up the values. I know this is hard to understand and visualize, so I'll try to explain what I mean:

     

    So our 2 bytes in binary are:

     

    10000000 00100000

     

    The bits that are part of the value, starting from the most significant to the least significant are:

     

          * 6 5 4 3 2 1 0   14 13 12 11 10 9 8 7
    Bits: 1 0 0 0 0 0 0 0 |  0  0  1  0  0 0 0 0

     

    * = Special bit, next byte part of this value.

     

    Now that we have the significance of each bit, this is how you convert significance to bit-value:

     

    2 ^ significance = bit-value

     

    To determine the final value, just add up all the bit values. So in this case we only have one bit set and the bit-value is 2 ^ 12 = 4096.

     

    So now we know that the first field value for "BytesPerCluster" is 4096.

     

    It doesn't make sense to do this calculation manually every time so I've built a simple VB.NET function to convert these.

     

    Here's the code:

     

    Public Shared Function FromWHSInt(ByVal TheBytes() As Byte) As UInt64
        Dim R As UInt64

        Dim ByteCount As Integer
        Dim SignificantBitCount As Integer

     

        For I As Integer = 0 To TheBytes.Length - 1
            If (TheBytes(I) And &H80) = &H80 Then
                '' One more byte
                ByteCount += 1
                SignificantBitCount += 7

                If I = TheBytes.Length - 1 Then
                    Throw New ArgumentException("More bytes expected")
                End If
            Else
                ByteCount += 1
                SignificantBitCount += 8
                Exit For
            End If
        Next

     

        SignificantBitCount -= 1

     

        Dim BitMask As Byte
        For I As Integer = ByteCount - 1 To 0 Step -1
            For J As Integer = 7 To 0 Step -1
                If I <> ByteCount - 1 AndAlso J = 7 Then
                    '' Last byte, skip insignificant bit
                    Continue For
                End If
                BitMask = CByte(&H1) << J
                If (TheBytes(I) And BitMask) = BitMask Then
                    R = R + CULng(2 ^ SignificantBitCount)
                End If
                SignificantBitCount -= 1
            Next
        Next

     

        Return R

     

    End Function

     

    This is of course very preliminary code.

     

     

    Sunday, July 13, 2008 11:42 PM
  •  

    I have not really examined the Records section so I'll leave that part for later. It would be nice if we can get this to the point of enough knowledge so that someone can build an external backup DB recovery/management tool. Based on what I've seen so far I don't think this is unreasonable.

     

    What do you guys think, would a backup management tool like this be useful?

     

    It would be nice to get some help from the WHS guys on this.

    Sunday, July 13, 2008 11:47 PM
  • Although you are making progress, I cannot imagine the WHS Team would be really interested. The consensus previously, is that WHS is in effect, a disaster recovery option. That is, if a hard drive fails, you can rebuild a Client. The loss of backups previously, have had the reply, just do another set of backups, as it's not  method of preserving ex-client backups for the long term.

    However, I would suggest you go to the Connect site and post your request there - with the details. Anything there, with other peoples votes, are looked at for further development  of WHS. Also, it will get more attention there, than here buried in the middle of a thread. 

    If you do so, post the link back here, for others to also vote on it if they feel it is important.

     

    Colin

    Monday, July 14, 2008 3:40 PM
  •  ColinWH wrote:

    The consensus previously, is that WHS is in effect, a disaster recovery option. That is, if a hard drive fails, you can rebuild a Client. The loss of backups previously, have had the reply, just do another set of backups, as it's not  method of preserving ex-client backups for the long term.



    I agree with that, I understood this limitations from the get go and I was perfectly fine using the system within those constraints. But over time, with computers getting old and replaced by newer ones it just happens to fall into that role naturally with time.

    On the MS connect note, I think that I have enough information to build my own consistency check and backup/recovery solution (probably in the form of an addin) for the backup database. I personally think it's a good idea, and I know I would find it very useful. I was just wondering if anyone here felt the same. While it would be helpful, the WHS team doesn't really need to get involved.
    Monday, July 14, 2008 11:05 PM
  • So, to put an end to this. Given my efforts above, armed with a hex editor, I was able to reconstruct the missing files and get my corrupt backup database to open and I am now able to pull files off.

    It would be a shame for the information discussed above to get lost so I'll probably be posting it somewhere else more permanent and also the promised 4th part.
    Monday, July 14, 2008 11:08 PM
  •  AlexPi wrote:

     

    I have not really examined the Records section so I'll leave that part for later. It would be nice if we can get this to the point of enough knowledge so that someone can build an external backup DB recovery/management tool. Based on what I've seen so far I don't think this is unreasonable.

     

    What do you guys think, would a backup management tool like this be useful?

     

    It would be nice to get some help from the WHS guys on this.

     

    Hi AlexPi,

     

    VERY nice work indeed!!!  And IMHO this is certainly interesting. Before digging into this to deep, there is a way to regain access to previous version of the backup database files (vss snaphots). (You can read about this in my post Fixing Database Inconsistency errors. In that post I also mention shadow copy functionality on WHS is more or less broken and that it will only work in certain conditions. However Kariya21, Ken Warren and me recently discovered a way to regain access to existing shadow copies.  This has not yet been published overhere, however it will be posted shortly, and I think this offers a very easy way to recover from corrupted client backup database.

     

    If you think you can add anything useful that can not be fixed with the previous versions method please let us know.

    Tuesday, July 15, 2008 5:32 AM
    Moderator
  •  brubber wrote:

    Hi AlexPi,

     

    VERY nice work indeed!!!  And IMHO this is certainly interesting. Before digging into this to deep, there is a way to regain access to previous version of the backup database files (vss snaphots). (You can read about this in my post Fixing Database Inconsistency errors. In that post I also mention shadow copy functionality on WHS is more or less broken and that it will only work in certain conditions. However Kariya21, Ken Warren and me recently discovered a way to regain access to existing shadow copies.  This has not yet been published overhere, however it will be posted shortly, and I think this offers a very easy way to recover from corrupted client backup database.

     

    If you think you can add anything useful that can not be fixed with the previous versions method please let us know.



    Cool! I'm looking forward to your VSS post. Is there any prerequisite to getting VSS working with a backup DB, or is it just on out of the box?

    I'm working on posting everything I discovered somewhere more permanent. I know this info is useful for recovery because it helped me, so hopefully someone else will find it useful too. I'll post the link here once that's up.
    Tuesday, July 15, 2008 1:36 PM
  •  AlexPi wrote:
    Cool! I'm looking forward to your VSS post. Is there any prerequisite to getting VSS working with a backup DB, or is it just on out of the box?

     

    On WHS RTM install from OEM disks vss is enabled by default, snapshot every 12h. On my own box I set this to 6h since I only have few changes / new files each day. On HPmediasmart server it's disabled by default, however pretty simple to enable by editing single registry key. I don't know for other OEM products.

     

    Vss snaphots are always for complete disk, so normally simpling sharing any folder on that disk will enable you to view and access that folders previous version. In WHS there are currently some problems with this in certain circumstances. (Link the faq from Ken Warren describing this in detail in  Fixing Database Inconsistency errors)

     

    Since backup DB is also just a bunch of files in a folder it will also work for backup DB. Conveniently WHS backup DB is stored in 4 GB or smaller files, so you only need to recover the files you need.

    Tuesday, July 15, 2008 1:53 PM
    Moderator
  • Just like I promised I've set up a blog documenting the backup DB format. Also, I've put up an amusing account of my data recovery perils.

    This will also be where I will be making tools available based on my knowledge.

    http://blog.whssafebackup.com/

    My ultimate goal is to make backing up on the home server bullet-proof. I intend to accomplish this with an addin I will make in the not too distant future. Watch the blog for updates.
    Sunday, July 27, 2008 10:17 PM
  • You've posted an interesting analysis of the backup database. I think you're going to find that true recovery from loss of many of the files results in the abandoning of almost all the backups in the database, unfortunately (i.e. the data files can't really be recreated).
    Monday, July 28, 2008 1:54 AM
    Moderator
  • Ken, actually I've already recreated a data file from my previous recovery experience, it had no real data obviously, but it was enough to get the database to be recognized.

    This was only a single data file, I agree that if you loose many data files your chances are slim.

    My approach to fixing this problem is different. Instead of creating a utility to fix a broken backup database, I want to eventually create an addin that will absolutely prevent the database from becoming corrupt. You can only accomplish this if you can perform an integrity check before backing up.
    Monday, July 28, 2008 12:28 PM
  • My point is that the loss of any one of the files that contains data (i.e. clusters) from your home computers has a high probability of destroying most, if not all, the backups that your server is storing. That's because there is only a single copy of those clusters in the database, and there will typically be multiple backups, possibly on multiple computers, referencing each of those clusters. I'm not sure how you intend to prevent the backup database from becoming corrupt in the first place; you need an excellent understanding of the mechanisms that result in a corrupt database to have a hope of that. But I certainly wish you luck in your endeavour.

    I've also called your efforts to the attention of the WHS team, who will (I'm sure) be interested in your work. Smile If you'll drop me an email (use the email address in my profile here in the forums; just remove the antispam device or I may not see it for days) I'll forward other contact information on to them, in case they're interested in following up directly.
    Monday, July 28, 2008 7:24 PM
    Moderator
  •  AlexPi wrote:
    Ken, actually I've already recreated a data file from my previous recovery experience, it had no real data obviously, but it was enough to get the database to be recognized.

    This was only a single data file, I agree that if you loose many data files your chances are slim.

    My approach to fixing this problem is different. Instead of creating a utility to fix a broken backup database, I want to eventually create an addin that will absolutely prevent the database from becoming corrupt. You can only accomplish this if you can perform an integrity check before backing up.

     

    To clarify: did you recreate a tombstone (4,096 bytes) or an actual data file (usually around 4 GB)?

    Monday, July 28, 2008 7:37 PM
    Moderator
  •  kariya21 wrote:

     AlexPi wrote:
    Ken, actually I've already recreated a data file from my previous recovery experience, it had no real data obviously, but it was enough to get the database to be recognized.

    This was only a single data file, I agree that if you loose many data files your chances are slim.

    My approach to fixing this problem is different. Instead of creating a utility to fix a broken backup database, I want to eventually create an addin that will absolutely prevent the database from becoming corrupt. You can only accomplish this if you can perform an integrity check before backing up.

     

    To clarify: did you recreate a tombstone (4,096 bytes) or an actual data file (usually around 4 GB)?



    I was able to recreate one 4GB data file and a Control.4096.dat file. The data file obviously did not have actual real data (I don't do magic), but it was enough to get the server to recognize and start working with the backup DB.
    Tuesday, July 29, 2008 1:29 AM
  •  AlexPi wrote:
    Ken, actually I've already recreated a data file from my previous recovery experience, it had no real data obviously, but it was enough to get the database to be recognized.

    This was only a single data file, I agree that if you loose many data files your chances are slim.

    My approach to fixing this problem is different. Instead of creating a utility to fix a broken backup database, I want to eventually create an addin that will absolutely prevent the database from becoming corrupt. You can only accomplish this if you can perform an integrity check before backing up.

     

    Hi AlexPi, I agree database integretity should be checked regularly. I do this manually by opening a few backup's each week. Actually I think WHS already does an integrity check before running backup or backup cleanup, however it doesn't seem to cover all possible issues.

     

    Personally I think a decent integrity check, combined with restore from a previous version should do the trick in most cases. In case you missed it (there so many threads) I have posted a method to regain access to previous version of files in WHS shares overhere.

    Wednesday, August 13, 2008 4:12 AM
    Moderator
  • So, I made a terrible mistake in not checking the validity and accessibility of my WHS backups before wiping out my wifes laptop hard drive.  Now, I can't restore anything since the mounting of the restore fails at 81%.  I downloaded and ran the WhsDbDataDump - 1.0.0 Build 6 BETA tool from AlexPi's website.  It worked somewhat, but most of the files are garbage.  Some work fine but most documents and images are unusable.  Are there any other options available?  AlexPi, would you be willing to share your sourcecode and let me take a crack at improving the results?  From the comments in the restore process, it sounds like it has something to do with compression and that most files aren't being decompressed when dumped from the database.
    Thursday, November 4, 2010 5:47 PM