locked
(solved) Restore failed with error 267 in WHS event log RRS feed

  • Question

  • Hi,

     

    I just had a catastrophic crash of my data drive, and I'm trying to restore my drive from the WHS backup. The restore (mount process) fails at 34% on the client PC with the error "Unable to mount drive".

    On the WHS server I see the following error in Event Viewer:
    Source: HomeServer
    Category: Backup
    Event ID: 267
    Client Backup server failed at d:\wssg_src\whs_pp3\shared\src\dienew\dienew.cpp(16)

    The home server logs on the client pc have the following error:

    <snipped>

    [12/12/2011 2:22:58 AM 1094] OnProgress: UI progress = 30%
    [12/12/2011 2:22:59 AM 12c4] PrepareBackupServerForMount: The backup server reported 34 percent complete
    [12/12/2011 2:22:59 AM 1094] OnProgress: UI progress = 31%
    [12/12/2011 2:23:00 AM 1094] OnProgress: UI progress = 31%
    [12/12/2011 2:23:00 AM 12c4] PrepareBackupServerForMount: The backup server reported 35 percent complete
    [12/12/2011 2:23:01 AM 1094] OnProgress: UI progress = 32%
    [12/12/2011 2:23:02 AM 12c4] PrepareBackupServerForMount: The backup server reported 36 percent complete
    [12/12/2011 2:23:02 AM 1094] OnProgress: UI progress = 32%
    [12/12/2011 2:23:03 AM 1094] OnProgress: UI progress = 32%
    [12/12/2011 2:23:04 AM 12c4] PrepareBackupServerForMount: The backup server reported 37 percent complete
    [12/12/2011 2:23:04 AM 1094] OnProgress: UI progress = 32%
    [12/12/2011 2:23:05 AM 1094] OnProgress: UI progress = 32%
    [12/12/2011 2:23:06 AM 12c4] PrepareBackupServerForMount: The backup server reported 38 percent complete
    [12/12/2011 2:23:06 AM 1094] OnProgress: UI progress = 33%
    [12/12/2011 2:23:07 AM 1094] OnProgress: UI progress = 33%
    [12/12/2011 2:23:08 AM 12c4] PrepareBackupServerForMount: The backup server reported 39 percent complete
    [12/12/2011 2:23:08 AM 1094] OnProgress: UI progress = 34%
    [12/12/2011 2:23:09 AM 1094] OnProgress: UI progress = 34%
    [12/12/2011 2:23:10 AM 1094] OnProgress: UI progress = 34%
    [12/12/2011 2:23:11 AM 12c4] Error 10054 receiving data from partner
    [12/12/2011 2:23:11 AM 12c4] Breaking TLS sesssion...
    [12/12/2011 2:23:11 AM 12c4] Read 0 bytes, hr=80072746
    [12/12/2011 2:23:11 AM 12c4] PrepareBackupServerForMount: Waiting for the backup server completed
    [12/12/2011 2:23:11 AM 12c4] ERROR: PrepareBackupServerForMount: unexpected message code from the server (0)
    [12/12/2011 2:23:11 AM 12c4] ERROR: Preparing Backup Server failed
    [12/12/2011 2:23:11 AM 12c4] Exiting thread...
    [12/12/2011 2:23:11 AM 1094] OnProgress: UI progress = 34%
    [12/12/2011 2:23:11 AM 1094] ERROR: RestoreHelper::Unable to mount drive. Error 11

    How do I get out of this bad state? I really cannot afford to lose this data. The drive has 10 years of photos, documents and music on it.





    • Edited by Aakash K Saturday, December 24, 2011 4:01 AM
    Monday, December 12, 2011 10:56 AM

Answers

  • I finally figured out what the issue was. The average user will *never* be able to come up with these steps to resolution, so I'm documenting my problem and

    resolution in the hope that it helps someone else in the future. I have to say that I'm extremely disappointed with the WHS team for the pathetic code quality

    of whsbackup.exe and the general WHS experience.

     

     

     

    What happened:

     

    - The data drive on my desktop machine was overwritten/nuked by new Intel Rapid Storage technology drivers (that is a whole different story). After a short

    amount of panic I figured I was ok since I would just restore the drive from WHS where the entire drive was being backed up.

     

    - I launched WHS connector and went to the "View Backups" tab for my desktop machine. I selected a recent backup for the drive in question and clicked "open".

     

    - WHS/WHS connector indicated the backup was being loaded. As I understood, the backup would show up as a network drive on my desktop once the rehydration of

    the backup was complete.

     

    - At 34% through the rehydration process (as indicated by the progress bar in the WHS Connector software), the mounting of the backup failed with an extremely

    generic "Unable to mount backup" error message. I tried this a couple more times and it consistently failed at 34%. I tried rebooting my desktop and the WHS,

    and this would consistently fail at the same point. Trying to mount a different (older) backup failed in exactly the same way.

     

    - At this point I was panicking. I was not using WHS folder duplication to store my files - relying on the backed up copies of the files to carry me through

    drive failure. In hindsight this strategy was clearly flawed.

     

    - I looked online on various forums (including this one) to determine if anybody else was seeing the same issue I was. Though there were many reports of backups

    failing at 81%, I could not find any instances of users having exactly the same issue I was having. I gathered various logs from the WHS box to try to figure

    out what was happening. That is when I made the series of posts above in the hope of finding help from some of the WHS experts on this forum.

     

    - The specific issue in my case was that WHSBackup.exe was crashing with an access violation ( c0000005 ) when trying to mount the backup, with error code 267.

    It looks like nobody else had hit this issue, so I suspected perhaps my hardware was bad. But running memtest86 from a USB drive at boot time indicated the

    memory was fine. (I have the VGA jumper cable for my HP EX495).

     

    - At this point I decided to try to recover my data using other means, so I backed up my entire d:\fs folder onto another hard drive.

     

    - I tried using the WHS db check and data dump tools from http://blog.whssafebackup.com/ to try to get my data. But these tools failed to get anything useful

    out of the backup database. Instead they increased my stress level by reporting the following error:

     

    ERROR 4007: Cannot check data                                                                

    File: F:\WHS DB\Backup\folders\{00008086-058D-4C89-AB57-A7F909A47AB4}\AK2.VolumeConfig.config.dat
    There was a problem verifying the data of file F:\WHS DB\Backup\folders\{00008086-058D-4C89-AB57-A7F909A47AB4}\AK2.VolumeConfig.config.dat for correctness.                Arithmetic operation resulted in an overflow.
    There was a problem detected with the backup database. This problem has caused data loss. It is possible that some portions of the database may still be accessible by the Windows Home Server database engine or by using specialized recovery tools. It is highly recommended to try and recover any crucial data and restore the database to a known good state as soon as possible. Data recovery can be attempted by using the Windows Home Server database engine or a specialized tool such as WhsDbDataDump.

     

    This error seemed to indicate that my WHS backup set itself was corrupt in some way, and that my data was now lost forever. However, I was slightly skeptical of this because of the following - every time the whsbackup.exe process started after crashing, it performs an integrity check on the backup database. The results of these integrity checks were in the logs, and seemed to indicate the backup set was OK. So I wasn't willing to give up yet.

     

    - When no help was forthcoming, it was time to get my hands dirty.

     

    What a friend and I did:

     

    - We hooked windbg.exe to whsbackup.exe, set up my symbol path to correctly point to Microsoft's symbol servers, and kicked of yet another attempt to mount the backup.

    - windbg.exe broke into the debugger. The stack trace indicated an out of memory exception deep inside whsbackup.exe when trying to allocate some kind of Tree structure. This was throwing.

    - A catch block somewhere caught the exception. It looked like it was trying to write an event to the error log with details of the out of memory exception. But it was allocating memory for the strings to write to the event log! The memory allocation failed and threw another out of memory exception.

    - WHS could not handle this second exception. The code at d:\wssg_src\whs_pp3\shared\src\dienew\dienew.cpp(16) intentionally killed the process by dereferencing a null pointer. This was the access violation we saw. This was caught by drwatson on the WHS box and a access violation stack trace was generated (what I pasted above).

     

    Here is the stack trace:

    WHSBackup!KillProcess+0xc7

    WHSBackup!FatalErrorImpl+0x21

    WHSBackup!operator new+0x2c

    WHSBackup!std::allocator<std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short>,_STL70> >::allocate+0x11:

    WHSBackup!std::vector<std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short>,_STL70>,std::allocator<std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short>,_STL70> > >::_Insert_n+0x92:

    WHSBackup!std::vector<std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short>,_STL70>,std::allocator<std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short>,_STL70> > >::insert+0x33:

    WHSBackup!std::vector<std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short>,_STL70>,std::allocator<std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short>,_STL70> > >::push_back+0x3a:

    WHSBackup!CClientBackupService::OnFatalApiError+0x7a:

    WHSBackup!FatalErrorImpl+0x16:

    WHSBackup!operator new+0x2c:

    WHSBackup!std::_Tree<std::_Tmap_traits<ClusterRangeMap::Key,ClusterRangeMap::Target,std::less<ClusterRangeMap::Key>,std::allocator<std::pair<ClusterRangeMap::Key const ,ClusterRangeMap::Target> >,0> >::_Buynode+0x10:

    WHSBackup!std::_Tree<std::_Tmap_traits<ClusterRangeMap::Key,ClusterRangeMap::Target,std::less<ClusterRangeMap::Key>,std::allocator<std::pair<ClusterRangeMap::Key const ,ClusterRangeMap::Target> >,0> >::_Insert+0x55:

    WHSBackup!std::_Tree<std::_Tmap_traits<ClusterRangeMap::Key,ClusterRangeMap::Target,std::less<ClusterRangeMap::Key>,std::allocator<std::pair<ClusterRangeMap::Key const ,ClusterRangeMap::Target> >,0> >::insert+0x106:

    WHSBackup!std::map<ClusterRangeMap::Key,ClusterRangeMap::Target,std::less<ClusterRangeMap::Key>,std::allocator<std::pair<ClusterRangeMap::Key const ,ClusterRangeMap::Target> > >::operator[]+0x5f:

    WHSBackup!ClusterRangeMap::Add+0x40:

    WHSBackup!RestoreOperation::PopulateClusterRangeMap+0x2f5:

    WHSBackup!RestoreOperation::DoRestore+0x2f9:

    WHSBackup!CSession::DoRestore+0x246:

    WHSBackup!CSession::ThreadMain+0x2b8:

    WHSBackup!CSession::ThreadMainStatic+0xd:

     

     

     

    So basically whsbackup.exe was running out of memory when trying to mount the backup. The following was also true:

     

    - WHSv1 is Windows Server 2003 32-bit.

    - My HP EX495 has 2GB of physical memory.

    - By default on 32-bit Windows OSes, each application process has a 2GB virtual address space. The OS reserves 2GB for itself.

     

    I also noticed that:

    - WHS does not have the /3GB flag in boot.ini, which would increase the process address space from 2GB to 3GB for any large address aware app running on the WHS. 

    - whsbackup.exe does not have the LARGE_ADDRESS_AWARE flag set in the process executable header

    - The default page file was set to 4GB

     

    In order to fix this:

    - I modified boot.ini and added the /3GB flag at the end

    - I edited whsbackup.exe and set the large address aware flag (try http://www.techpowerup.com/forums/showthread.php?t=112556 )

    - I increased the size of the page file to 8GB.

    - Rebooted.

     

    Crossed my fingers and tried to mount the backup again. I was carefully monitoring whsbackup.exe's memory consumption. I saw it increase all the way to 2.8GB, so it looked like increasing it's virtual address space helped. I was able to mount the backup, and copy my files off.

     

    Hopefully this will help someone out. For what it's worth, my backup was about 370GB or so. It would have been nice if the WHS team had tested mounting of large backups on machines with 2GB physical memory in the default WHS configuration.

     

     





    • Marked as answer by Aakash K Saturday, December 24, 2011 4:00 AM
    • Edited by Aakash K Saturday, December 24, 2011 4:15 AM
    Saturday, December 24, 2011 4:00 AM

All replies

  • Trying to restore using the ClientRestoreWizard results in a failure on the client and the same error on the Windows home server:

    Event Type: Error
    Event Source: HomeServer
    Event Category: Backup
    Event ID: 267
    Date:  12/12/2011
    Time:  7:50:30 PM
    User:  N/A
    Computer: WHS
    Description:
    Client Backup server failed at d:\wssg_src\whs_pp3\shared\src\dienew\dienew.cpp(16)

    For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

    Any help would be appreciated. Are there server logs on the WHS which might indicate what the problem is with the "Client Backup server" ?

    I'd like to add that I have run checkdisk on the WHS on all drives (there are three data drives) and no issues were reported.


    • Edited by Aakash K Tuesday, December 13, 2011 4:07 AM
    Tuesday, December 13, 2011 4:04 AM
  • The crash is an access violation in whsbackup.exe:

     

    Application exception occurred:
            App: C:\Program Files\Windows Home Server\whsbackup.exe (pid=5736)
            When: 12/12/2011 @ 20:56:08.765
            Exception number: c0000005 (access violation)

    *----> System Information <----*
            Computer Name: WHS
            User Name: SYSTEM
            Terminal Session Id: 0
            Number of Processors: 2
            Processor Type: x86 Family 6 Model 23 Stepping 10
            Windows Version: 5.2
            Current Build: 3790
            Service Pack: 2
            Current Type: Multiprocessor Free
            Registered Organization: Windows Home Server
            Registered Owner: Windows Home Server

    *----> Task List <----*
       0 System Process
       4 System
     360 smss.exe
     416 csrss.exe
     440 winlogon.exe
     488 services.exe
     500 lsass.exe
     676 svchost.exe
     748 svchost.exe
     820 svchost.exe
     848 svchost.exe
     872 svchost.exe
    1008 spoolsv.exe
    1044 msdtc.exe
    1140 mDNSResponder.exe
    1172 CrashPlanService.exe
    1212 svchost.exe
    1244 firefly.exe
    1332 inetinfo.exe
    1376 llssrv.exe
    1456 p4s.exe
    1504 svchost.exe
    1608 sbscrexe.exe
    1648 SiHbaWakeupService.exe
    1664 svchost.exe
    1724 svchost.exe
    1756 twonkymedia.exe
    1836 svchost.exe
    1852 vds.exe
    1920 vssvc.exe
    1944 WLIDSVC.EXE
    2076 SearchIndexer.exe
    2116 TwonkyMediaServer.exe
    2156 HpmssService.exe
    2452 WLIDSvcM.exe
    2504 wmiprvse.exe
    2568 mqsvc.exe
    2624 qsm.exe
    2844 svchost.exe
    2868 whsarch.exe
    3036 TransportService.exe
    3088 wmccds.exe
    3152 cqvSvc.exe
    3488 mqtgsvc.exe
    3504 pdl.exe
    3584 svchost.exe
    3608 dmadmin.exe
    3748 alg.exe
    4040 w3wp.exe
     408 demigrator.exe
     696 w3wp.exe
    5040 csrss.exe
    5068 winlogon.exe
    5216 rdpclip.exe
    5284 homeserverconsole.exe
    5620 svchost.exe
    5792 PresentationHost.exe
    6084 csrss.exe
    2428 winlogon.exe
    2252 rdpclip.exe
    5376 Explorer.EXE
    1716 CrashPlanTray.exe
    4180 WindowsSearch.exe
    2888 mmc.exe
    5736 whsbackup.exe
     788 w3wp.exe
    4864 drwtsn32.exe

    *----> Module List <----*
    0000000000400000 - 000000000040b000: C:\Program Files\Windows Home Server\custsat.dll
    0000000000760000 - 0000000000775000: C:\Program Files\Windows Home Server\WHSNotificationFactory.dll
    0000000000780000 - 00000000007a5000: C:\Program Files\Windows Home Server\WHSNotificationSource.dll
    0000000000860000 - 0000000000883000: C:\Program Files\Windows Home Server\PartnerManager.dll
    00000000009f0000 - 0000000000cb5000: C:\WINDOWS\system32\xpsp2res.dll
    0000000000d50000 - 0000000000d5f000: C:\Program Files\Windows Home Server\TransportServiceProxy.dll
    0000000001000000 - 0000000001083000: C:\Program Files\Windows Home Server\whsbackup.exe
    000000004e7c0000 - 000000004e81d000: C:\WINDOWS\WinSxS\x86_Microsoft.Windows.WinHTTP_6595b64144ccf1df_5.1.3790.4584_x-ww_FE3453F4\WINHTTP.dll
    000000005f270000 - 000000005f2ca000: C:\WINDOWS\system32\hnetcfg.dll
    0000000068000000 - 0000000068035000: C:\WINDOWS\system32\rsaenh.dll
    0000000068100000 - 0000000068127000: C:\WINDOWS\system32\dssenh.dll
    0000000071ae0000 - 0000000071ae8000: C:\WINDOWS\System32\wshtcpip.dll
    0000000071b20000 - 0000000071b61000: C:\WINDOWS\system32\mswsock.dll
    0000000071bf0000 - 0000000071bf8000: C:\WINDOWS\system32\WS2HELP.dll
    0000000071c00000 - 0000000071c17000: C:\WINDOWS\system32\WS2_32.dll
    0000000071c40000 - 0000000071c97000: C:\WINDOWS\system32\NETAPI32.dll
    0000000075da0000 - 0000000075e5d000: C:\WINDOWS\system32\SXS.DLL
    0000000076190000 - 00000000761a2000: C:\WINDOWS\system32\MSASN1.dll
    00000000761b0000 - 0000000076243000: C:\WINDOWS\system32\CRYPT32.dll
    00000000766e0000 - 00000000766ec000: C:\WINDOWS\system32\cryptdll.dll
    0000000076750000 - 0000000076778000: C:\WINDOWS\system32\schannel.dll
    0000000076920000 - 00000000769e2000: C:\WINDOWS\system32\USERENV.dll
    0000000076b70000 - 0000000076b7b000: C:\WINDOWS\system32\PSAPI.DLL
    0000000076c90000 - 0000000076cb7000: C:\WINDOWS\system32\msv1_0.dll
    0000000076cf0000 - 0000000076d0a000: C:\WINDOWS\system32\IPHLPAPI.DLL
    0000000076ed0000 - 0000000076efa000: C:\WINDOWS\system32\DNSAPI.dll
    0000000076f50000 - 0000000076f63000: C:\WINDOWS\system32\Secur32.dll
    0000000077010000 - 00000000770d6000: C:\WINDOWS\system32\COMRes.dll
    0000000077380000 - 0000000077411000: C:\WINDOWS\system32\USER32.dll
    0000000077420000 - 0000000077523000: C:\WINDOWS\WinSxS\x86_Microsoft.Windows.Common-Controls_6595b64144ccf1df_6.0.3790.4770_x-ww_05FDF087\comctl32.dll
    0000000077670000 - 00000000777a9000: C:\WINDOWS\system32\ole32.dll
    00000000777b0000 - 0000000077833000: C:\WINDOWS\system32\CLBCatQ.DLL
    0000000077b90000 - 0000000077b98000: C:\WINDOWS\system32\VERSION.dll
    0000000077ba0000 - 0000000077bfa000: C:\WINDOWS\system32\msvcrt.dll
    0000000077c00000 - 0000000077c49000: C:\WINDOWS\system32\GDI32.dll
    0000000077c50000 - 0000000077cf0000: C:\WINDOWS\system32\RPCRT4.dll
    0000000077e40000 - 0000000077f42000: C:\WINDOWS\system32\kernel32.dll
    000000007c800000 - 000000007c8c3000: C:\WINDOWS\system32\ntdll.dll
    000000007c8d0000 - 000000007d0cf000: C:\WINDOWS\system32\SHELL32.dll
    000000007d0e0000 - 000000007d16b000: C:\WINDOWS\system32\OLEAUT32.dll
    000000007d180000 - 000000007d1d2000: C:\WINDOWS\system32\SHLWAPI.dll
    000000007d1e0000 - 000000007d27c000: C:\WINDOWS\system32\ADVAPI32.dll

    <snipped other threads>

    *----> State Dump for Thread Id 0x165c <----*

    eax=00000000 ebx=00000002 ecx=77e769aa edx=7c82847c esi=00000000 edi=00000001
    eip=01063484 esp=0069f36c ebp=0069f694 iopl=0         nv up ei pl zr na po nc
    cs=001b  ss=0023  ds=0023  es=0023  fs=003b  gs=0000             efl=00010246

    function: whsbackup
            01063469 fd               std
            0106346a ffff             ???
            0106346c 50               push    eax
            0106346d e8a51c0000       call    whsbackup+0x65117 (01065117)
            01063472 8d8528fdffff     lea     eax,[ebp-0x2d8]
            01063478 50               push    eax
            01063479 ff1584110001     call    dword ptr [whsbackup+0x1184 (01001184)]
            0106347f 3bc6             cmp     eax,esi
            01063481 5e               pop     esi
            01063482 7507             jnz     whsbackup+0x6348b (0106348b)
    FAULT ->01063484 a100000000       mov     eax,[00000000]    ds:0023:00000000=????????
            01063489 eb0f             jmp     whsbackup+0x6349a (0106349a)
            0106348b 6a03             push    0x3
            0106348d ff15f8110001     call    dword ptr [whsbackup+0x11f8 (010011f8)]
            01063493 50               push    eax
            01063494 ff1540120001     call    dword ptr [whsbackup+0x1240 (01001240)]
            0106349a 8b4dfc           mov     ecx,[ebp-0x4]
            0106349d 33cd             xor     ecx,ebp
            0106349f e8e1b2fcff       call    whsbackup+0x2e785 (0102e785)
            010634a4 c9               leave
            010634a5 c20800           ret     0x8

    *----> Stack Back Trace <----*
    ChildEBP RetAddr  Args to Child             
    WARNING: Stack unwind information not available. Following frames may be wrong.
    0069f694 010635d7 010042c8 00000010 0069f6c0 whsbackup+0x63484
    0069f6a4 0102da05 010042c8 00000010 0100432c whsbackup+0x635d7
    0069f6c0 01017e0b 00000038 0069f724 01021076 whsbackup+0x2da05
    0069f6cc 01021076 00000002 c4628950 0069f794 whsbackup+0x17e0b
    0069f724 0102121a 00ebbe3c 00000001 0069f778 whsbackup+0x21076
    0069f740 010213f8 0069f764 00ebbe3c 0069f778 whsbackup+0x2121a
    0069f75c 0102158a 0069f778 c46289c4 00ec48c4 whsbackup+0x213f8
    0069f7b0 010635cc 0100432c 8007000e 0069f7dc whsbackup+0x2158a
    0069f7c0 0102da05 010042c8 00000010 0100432c whsbackup+0x635cc
    0069f7dc 01023375 00000038 00ec48c4 5f25efc8 whsbackup+0x2da05
    0069f7f0 01023681 00ec4b78 5f25efc8 00ec4b78 whsbackup+0x23375
    0069f86c 01023c9c 0069f8f4 00000000 5f25efc8 whsbackup+0x23681
    0069f898 0102419c 0069f8f4 00ec4b78 0069f8b8 whsbackup+0x23c9c
    0069f8ec 010241ed 0069f910 00ec4890 0069f950 whsbackup+0x2419c
    0069f920 010245c5 125e1634 00000000 00000001 whsbackup+0x241ed
    0069fb1c 010259e7 c4628358 0069fe3c 77ea37ee whsbackup+0x245c5
    0069fd2c 0102ad41 0069fe14 c4628038 00ec4e9c whsbackup+0x259e7
    0069fe4c 0102cee0 c46281c4 00000000 00000000 whsbackup+0x2ad41
    0069ffb0 0102d072 0069ffec 77e6482f 00ec4e80 whsbackup+0x2cee0
    0069ffb8 77e6482f 00ec4e80 00000000 00000000 whsbackup+0x2d072
    0069ffec 00000000 0102d065 00ec4e80 00000000 kernel32!GetModuleHandleA+0xdf

    *----> Raw Stack Dump <----*
    000000000069f36c  25 00 00 c0 01 00 00 00 - 00 00 00 00 d7 35 06 01  %............5..
    000000000069f37c  00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  ................
    000000000069f38c  00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  ................
    000000000069f39c  00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  ................
    000000000069f3ac  00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  ................
    000000000069f3bc  6c f3 69 00 c4 f3 69 00 - 00 00 00 00 00 00 00 00  l.i...i.........
    000000000069f3cc  00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  ................
    000000000069f3dc  00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  ................
    000000000069f3ec  00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  ................
    000000000069f3fc  00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  ................
    000000000069f40c  00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  ................
    000000000069f41c  00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  ................
    000000000069f42c  00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  ................
    000000000069f43c  00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  ................
    000000000069f44c  00 00 00 00 00 00 00 00 - 3b 00 00 00 23 00 00 00  ........;...#...
    000000000069f45c  23 00 00 00 01 00 00 00 - 00 00 00 00 02 00 00 00  #...............
    000000000069f46c  00 00 00 00 00 00 00 00 - c4 f3 69 00 a4 f6 69 00  ..........i...i.
    000000000069f47c  d7 35 06 01 1b 00 00 00 - 02 02 00 00 9c f6 69 00  .5............i.
    000000000069f48c  23 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  #...............
    000000000069f49c  00 00 00 00 00 00 00 00 - 00 00 00 00 00 00 00 00  ................

     

    Tuesday, December 13, 2011 4:23 PM
  • I finally figured out what the issue was. The average user will *never* be able to come up with these steps to resolution, so I'm documenting my problem and

    resolution in the hope that it helps someone else in the future. I have to say that I'm extremely disappointed with the WHS team for the pathetic code quality

    of whsbackup.exe and the general WHS experience.

     

     

     

    What happened:

     

    - The data drive on my desktop machine was overwritten/nuked by new Intel Rapid Storage technology drivers (that is a whole different story). After a short

    amount of panic I figured I was ok since I would just restore the drive from WHS where the entire drive was being backed up.

     

    - I launched WHS connector and went to the "View Backups" tab for my desktop machine. I selected a recent backup for the drive in question and clicked "open".

     

    - WHS/WHS connector indicated the backup was being loaded. As I understood, the backup would show up as a network drive on my desktop once the rehydration of

    the backup was complete.

     

    - At 34% through the rehydration process (as indicated by the progress bar in the WHS Connector software), the mounting of the backup failed with an extremely

    generic "Unable to mount backup" error message. I tried this a couple more times and it consistently failed at 34%. I tried rebooting my desktop and the WHS,

    and this would consistently fail at the same point. Trying to mount a different (older) backup failed in exactly the same way.

     

    - At this point I was panicking. I was not using WHS folder duplication to store my files - relying on the backed up copies of the files to carry me through

    drive failure. In hindsight this strategy was clearly flawed.

     

    - I looked online on various forums (including this one) to determine if anybody else was seeing the same issue I was. Though there were many reports of backups

    failing at 81%, I could not find any instances of users having exactly the same issue I was having. I gathered various logs from the WHS box to try to figure

    out what was happening. That is when I made the series of posts above in the hope of finding help from some of the WHS experts on this forum.

     

    - The specific issue in my case was that WHSBackup.exe was crashing with an access violation ( c0000005 ) when trying to mount the backup, with error code 267.

    It looks like nobody else had hit this issue, so I suspected perhaps my hardware was bad. But running memtest86 from a USB drive at boot time indicated the

    memory was fine. (I have the VGA jumper cable for my HP EX495).

     

    - At this point I decided to try to recover my data using other means, so I backed up my entire d:\fs folder onto another hard drive.

     

    - I tried using the WHS db check and data dump tools from http://blog.whssafebackup.com/ to try to get my data. But these tools failed to get anything useful

    out of the backup database. Instead they increased my stress level by reporting the following error:

     

    ERROR 4007: Cannot check data                                                                

    File: F:\WHS DB\Backup\folders\{00008086-058D-4C89-AB57-A7F909A47AB4}\AK2.VolumeConfig.config.dat
    There was a problem verifying the data of file F:\WHS DB\Backup\folders\{00008086-058D-4C89-AB57-A7F909A47AB4}\AK2.VolumeConfig.config.dat for correctness.                Arithmetic operation resulted in an overflow.
    There was a problem detected with the backup database. This problem has caused data loss. It is possible that some portions of the database may still be accessible by the Windows Home Server database engine or by using specialized recovery tools. It is highly recommended to try and recover any crucial data and restore the database to a known good state as soon as possible. Data recovery can be attempted by using the Windows Home Server database engine or a specialized tool such as WhsDbDataDump.

     

    This error seemed to indicate that my WHS backup set itself was corrupt in some way, and that my data was now lost forever. However, I was slightly skeptical of this because of the following - every time the whsbackup.exe process started after crashing, it performs an integrity check on the backup database. The results of these integrity checks were in the logs, and seemed to indicate the backup set was OK. So I wasn't willing to give up yet.

     

    - When no help was forthcoming, it was time to get my hands dirty.

     

    What a friend and I did:

     

    - We hooked windbg.exe to whsbackup.exe, set up my symbol path to correctly point to Microsoft's symbol servers, and kicked of yet another attempt to mount the backup.

    - windbg.exe broke into the debugger. The stack trace indicated an out of memory exception deep inside whsbackup.exe when trying to allocate some kind of Tree structure. This was throwing.

    - A catch block somewhere caught the exception. It looked like it was trying to write an event to the error log with details of the out of memory exception. But it was allocating memory for the strings to write to the event log! The memory allocation failed and threw another out of memory exception.

    - WHS could not handle this second exception. The code at d:\wssg_src\whs_pp3\shared\src\dienew\dienew.cpp(16) intentionally killed the process by dereferencing a null pointer. This was the access violation we saw. This was caught by drwatson on the WHS box and a access violation stack trace was generated (what I pasted above).

     

    Here is the stack trace:

    WHSBackup!KillProcess+0xc7

    WHSBackup!FatalErrorImpl+0x21

    WHSBackup!operator new+0x2c

    WHSBackup!std::allocator<std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short>,_STL70> >::allocate+0x11:

    WHSBackup!std::vector<std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short>,_STL70>,std::allocator<std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short>,_STL70> > >::_Insert_n+0x92:

    WHSBackup!std::vector<std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short>,_STL70>,std::allocator<std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short>,_STL70> > >::insert+0x33:

    WHSBackup!std::vector<std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short>,_STL70>,std::allocator<std::basic_string<unsigned short,std::char_traits<unsigned short>,std::allocator<unsigned short>,_STL70> > >::push_back+0x3a:

    WHSBackup!CClientBackupService::OnFatalApiError+0x7a:

    WHSBackup!FatalErrorImpl+0x16:

    WHSBackup!operator new+0x2c:

    WHSBackup!std::_Tree<std::_Tmap_traits<ClusterRangeMap::Key,ClusterRangeMap::Target,std::less<ClusterRangeMap::Key>,std::allocator<std::pair<ClusterRangeMap::Key const ,ClusterRangeMap::Target> >,0> >::_Buynode+0x10:

    WHSBackup!std::_Tree<std::_Tmap_traits<ClusterRangeMap::Key,ClusterRangeMap::Target,std::less<ClusterRangeMap::Key>,std::allocator<std::pair<ClusterRangeMap::Key const ,ClusterRangeMap::Target> >,0> >::_Insert+0x55:

    WHSBackup!std::_Tree<std::_Tmap_traits<ClusterRangeMap::Key,ClusterRangeMap::Target,std::less<ClusterRangeMap::Key>,std::allocator<std::pair<ClusterRangeMap::Key const ,ClusterRangeMap::Target> >,0> >::insert+0x106:

    WHSBackup!std::map<ClusterRangeMap::Key,ClusterRangeMap::Target,std::less<ClusterRangeMap::Key>,std::allocator<std::pair<ClusterRangeMap::Key const ,ClusterRangeMap::Target> > >::operator[]+0x5f:

    WHSBackup!ClusterRangeMap::Add+0x40:

    WHSBackup!RestoreOperation::PopulateClusterRangeMap+0x2f5:

    WHSBackup!RestoreOperation::DoRestore+0x2f9:

    WHSBackup!CSession::DoRestore+0x246:

    WHSBackup!CSession::ThreadMain+0x2b8:

    WHSBackup!CSession::ThreadMainStatic+0xd:

     

     

     

    So basically whsbackup.exe was running out of memory when trying to mount the backup. The following was also true:

     

    - WHSv1 is Windows Server 2003 32-bit.

    - My HP EX495 has 2GB of physical memory.

    - By default on 32-bit Windows OSes, each application process has a 2GB virtual address space. The OS reserves 2GB for itself.

     

    I also noticed that:

    - WHS does not have the /3GB flag in boot.ini, which would increase the process address space from 2GB to 3GB for any large address aware app running on the WHS. 

    - whsbackup.exe does not have the LARGE_ADDRESS_AWARE flag set in the process executable header

    - The default page file was set to 4GB

     

    In order to fix this:

    - I modified boot.ini and added the /3GB flag at the end

    - I edited whsbackup.exe and set the large address aware flag (try http://www.techpowerup.com/forums/showthread.php?t=112556 )

    - I increased the size of the page file to 8GB.

    - Rebooted.

     

    Crossed my fingers and tried to mount the backup again. I was carefully monitoring whsbackup.exe's memory consumption. I saw it increase all the way to 2.8GB, so it looked like increasing it's virtual address space helped. I was able to mount the backup, and copy my files off.

     

    Hopefully this will help someone out. For what it's worth, my backup was about 370GB or so. It would have been nice if the WHS team had tested mounting of large backups on machines with 2GB physical memory in the default WHS configuration.

     

     





    • Marked as answer by Aakash K Saturday, December 24, 2011 4:00 AM
    • Edited by Aakash K Saturday, December 24, 2011 4:15 AM
    Saturday, December 24, 2011 4:00 AM
  • Hi,

    I found myself in an equal desaster. On my PC one harddisk failed, it seemed that the motor stopped and the drive stopped running. I did not worry about that for i had my WHS with the backup. But when I tryed to restore the content of the backup to the replaced HDD I got the message, that the system is unable to mount the backup.

    Imagine how glad I was, to find your solution in the forum.

    I tried your hints, modified the boot.ini, edited whsbackup, increased the size of the page file and rebooted.

    The WHS then found itself in a permanet loop, it rebooted, seemed to fail and rebooted again.

    So I baught me the VGA-Adapter with PS/2-ports and assembled it to the server. On the screen I saw, that my guess was right. I booted the server in the save-mode and deleted my modification in the boot.ini. The next reboot worked, but the restoring of the the backup stopped.

    Probably I made a mistake when modifying the boot.ini

    Can you please post the whole content of this file or at least the line with the modification?

    Harry

    Wednesday, July 4, 2012 7:55 PM
  • "So basically whsbackup.exe was running out of memory when trying to mount the backup."

    If it's not a stupid question, do you think increasing the RAM on the server from 2Gb to 3Gb would achieve the same outcome? Having the same problem. Thx.

    Sunday, August 12, 2012 9:00 AM
  • The problem is usually a corrupted backup database, which can have multiple reasons (usually a reboot of the server at wrong time due to power outage or system crash or issues with file system due to damaged sectors etc on the disk).

    Sometimes repairing the database using the console works, often not.

    So if the backup database is damaged, usually increasing the memory or hacking files won't help a lot, especially if you don't exactly know what you are doing or how to deal with complications. Imagine a damaged file missing the end, so the restore process assumes the file is going on - and in reality it is not. So more and more memory is consumed by trying to load something beyond the real end of the file ...

    Best greetings from Germany
    Olaf

    Monday, August 13, 2012 12:47 PM
    Moderator
  • Thanks Olaf. For the sake of a few dollars, I'm going to increase the memory of my server from 2Gb to 3gb, just to test my (v. v. long shot!) theory. Naturally I'll report back...

    When this doesn't work - which is likely to be the case - I'll the give the instructions (above) at try, namely:

    ---------------------------------

    In order to fix this:

    - I modified boot.ini and added the /3GB flag at the end

    - I edited whsbackup.exe and set the large address aware flag (try http://www.techpowerup.com/forums/showthread.php?t=112556 )

    - I increased the size of the page file to 8GB.

    - Rebooted.

    Friday, August 17, 2012 8:04 AM
  • Apologies for the super-late response Harry. I didn't get any notifications about this thread. 

    I followed the instructions at http://technet.microsoft.com/en-us/library/bb124810(v=exchg.65).aspx to modify my boot.ini. I don't have the actual contents of my boot.ini any more since I stopped using WHS after this disaster. 

    Aakash

    Thursday, August 30, 2012 6:25 AM
  • Any luck? FWIW you should check that WhsBackup.exe is indeed running out of memory - do you see event ID 267 in your WHS event logs when you try to mount the backup?
    • Edited by Aakash K Thursday, August 30, 2012 6:27 AM more details
    Thursday, August 30, 2012 6:25 AM
  • Hi Aakash,

    Sadly your solution didn't work for me. Been working, trying, restarting all afternoon, but to no avail. Always, always, always fails at 81%.

    This alternative solution works for some, but not me: http://kelvyntaylor.blogspot.com.au/2010/11/fixing-cannot-mount-backup-problem-in.html

    My last hope I think might be WhsDbDataDump (if I can work out how to use it): http://blog.whssafebackup.com/

    Fyi:

    * On my WHS, Event Viewer > HomeServer gives an Event ID of 266, not 267.

    * I watch whsbackup.exe on the server, and it gets to about 153Mb of RAM usage at the 81% failure point.

    * On the PC where I'm trying to perform the restore:

    Log Name:      Application
    Source:        Application Error
    Date:          1/09/2012 7:43:40 PM
    Event ID:      1000
    Task Category: (100)
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      Inspiron-1545
    Description:
    Faulting application name: MountBackup.exe, version: 6.0.3436.0, time stamp: 0x4d2b680d
    Faulting module name: MountBackup.exe, version: 6.0.3436.0, time stamp: 0x4d2b680d
    Exception code: 0xc0000005
    Fault offset: 0x0000000000030f3a
    Faulting process id: 0x14bc
    Faulting application start time: 0x01cd88261c1f9789
    Faulting application path: C:\Program Files\Windows Home Server\MountBackup.exe
    Faulting module path: C:\Program Files\Windows Home Server\MountBackup.exe
    Report Id: 8257c322-f419-11e1-96f5-c44619eb9bae



    I think my next home server will be a dedicated NAS. I'm over the DIY home server thing! http://www.synology.com/products/product.php?product_name=DS412%2B&lang=us
    Saturday, September 1, 2012 10:05 AM