locked
Backup Cleanup Fails -- Insufficient RAM? RRS feed

  • General discussion

  • My HP MediaSmart Server cannot complete a Backup Cleanup.  It gets roughly 10% through the process and then displays an error message.  The title bar for the error msg window says "Windows Home Server Console cannot complete the task," and inside that msg window it says, "An error in the backup service is preventing the cleanup operation."  Other than this problem with the Backup Cleanup, my server seems to work fine.

    Using CHKDSK on C: and D: reports no errors.  Then I looked in the Event Viewer under the HomeServer subcategory, and for each failed Backup Cleanup, I found a pair of errors: (A) Insufficient RAM for sorting, followed by (B) Client Backup server failed at d:\wssg_src\whs_pp3\qhs\src\backup\util\sort.cpp(647)."

    Each time the Backup Cleanup failed, I closed the error message displayed by the Home Server Console and the recovery completes in mere minutes with no permanent ill effects that I can detect; all my client systems get a green checkmark.  I never had this problem before and wonder what the cause might be.

    System Info: HP MediaSmart Server, running the AMD Sempron Processor 3400+, 1.8 GHz, 480 MB RAM, with Virtual Memory (PageFile) set to 720 MB, and PowerPack 3 is installed.  WHS software versions are shown below.  C: has 11 GB free out of 20.

    In case it matters: I have seven client computers saved on my server; there are three disk drives installed (including the system disk) and I have 55% of server storage space available.  As mentioned above, CHKDSK reports no errors. 

    Windows Home Server Console: 6.0.2423.0
    Windows Home Server Backup & Restore: 6.0.2423.0
    Windows Home Server Drive Extender: 6.0.2423.0
    Windows Home Server Remote Access: 6.0.3436.0
    Windows Home Server Storage Manager: 6.0.3039.0

    There is one other odd thing that I never noticed until now: When I RDC to the server and display the properties for D:, the Used Space, Free Space, and Capacity figures aren't making sense; this includes gibberish characters where some of the Used Space figures should appear.  Perhaps this is normal for D:?

    Thanks in advance for whatever assistance you can provide.

    Thursday, May 5, 2011 2:59 AM

All replies

  • Hi,

    please check the event logs of your server, if there are any errors or warnings pointing to file system or disk issues.

    It could well be, that a part of the backup database is corrupted (thus causing the Backup service to crash), and a repair may or may not fix this. In worst case you may have to delete the entire backup database and start over with client backups.

    This being said, I recommend to upgrade the memory in your Mediasmart. The original 512 MB RAM are indeed the absolute minimum and make really working with WHS a bit painful.

    While 2 GB usually works, some users (including me) have experienced instabilities - random system freezes and network floodings each few days after such an upgrade, which did not happen for me after downgrading to 1 GB RAM.

    Best greetings from Germany

    Olaf

    Thursday, May 5, 2011 7:21 AM
    Moderator
  • As Olaf recommended remotely log into your WHS using RDP, and check the event viewer.

    You could also take a look at the log from the last run of chkdsk on your D drive...  go to:

    C:\Documents and Settings\All Users\Application Data\Microsoft\Windows Home Server\logs\

    and take a look at the file:

    chkdsk_D_.log

    Look to see if there are and bad blocks reported...  That will be the last run of chkdsk on your D drive which should have run around midnight the night before.

     

    Once you've confirmed that you do not have a disk error... and worst case you have to delete all of your backups, you can do that using the following method:

    http://www.clarkezone.net/default.aspx?id=532d3d54-b935-40e7-b020-beb0ca1d653a

    Thursday, May 5, 2011 2:41 PM
  • Olaf and Activoice,

    I very much appreciate your replies.  However, mentioning the possibility of having to delete my entire backup database is a little worrisome to me.

    I realize that a certain/positive diagnosis from where you're sitting is not possible.  However, please consider the following: (1) my CHKDSKs for C: and D: come back clean, (2) the logs don't indicate other issues (except insufficient RAM), (3) the backups and restores continue to work fine, and (4) I'm not having any other WHS problems.  Considering these things, do you feel a corrupt database is likely?  That's all I'm really asking.

    In the mean time, my cleanup can wait until I've installed more RAM.  But thanks again.

    Thursday, May 5, 2011 10:50 PM
  • Sometimes such error messages about not enough RAM can be irritating and refer to something else. Like not enough free disk space on drive C: or D:, or problems finding end of file in one of the backup database files (which would not affect the ability to restore something). But as said, adding more RAM on this appliance is definitively worth it.

    Please be aware, that the Backup database by concept is not seen as an archive, but as a tool to quickly restore disks or files in case of an immediate failure. Usually you will be able to backup the clients again if the backup database breaks. If you need to use it as kind of archive, use the Add On WHS Backup Database Backup to create a copy on an external disk from time to time, since it is so easy to loose the database (one failing disk in server or a file system issue maybe due to power outage is enough).

    Best greetings from Germany
    Olaf

    Friday, May 6, 2011 7:38 AM
    Moderator
  • Use the Add On WHS Backup Database Backup to create a copy on an external disk from time to time, since it is so easy to loose the database (one failing disk in server or a file system issue maybe due to power outage is enough).

    Olaf, sir ... Thank you again.  I had forgotten about the ability to backup the WHS server.  Somewhere I have a link already saved for the most popular WHS add-ons.  I meant to install one of the backup add-ons, but I never got around to it.  [Not a very good excuse, I know.]  Since I have a 2 TByte external hard drive, I'll start doing backups just as you suggested.

    Yes, I found the link for the most popular WHS add-onsIs there a particular add-on you'd like to recommend for backing up my MediaSmart Server?  [Actually, I eventually understood what you meant.  I noticed there's a backup add-on called "WHS Backup Database Backup."  I wish it were on the "most popular" list, so I suppose I'll have to research this further before actually trying it.]

    By the way, I'll just mention that I use a UPS units with my MediaSmart Server and with all my clients.  Also, I'm not sure whether you'd approve, but since I'm not using any WHS features other than for backups, and since I do backups just every other day, I don't turn my WHS server on until I'm about to perform my client backups.

    Thank you again and have a good weekend!


    • Edited by TonyM408 Saturday, May 7, 2011 6:14 AM Improved reply
    Friday, May 6, 2011 5:26 PM
  • Has anyone successfully resolved this problem without wiping out the backup DB and losing all your backups?  I have precisely the problem described above and get the two stated errors (insufficient RAM followed by the failure in sort.cpp(647)) in the WHS event log when I attempt backup cleanup.  Everything else works fine, chkdsk is clean, backups succeed every night, database cleanup succeeds and reports no errors, but backup cleanup always fails and thus it appears my backup space is always growing and I can't reclaim space by throwing away old backups.  Turning on verbose logging for the backup service did not reveal any more information.  The backup log always indicates that it is starting phase 2 and then the next log entry is the failure:

    [11/9/2013 11:37:12 AM 1020] Cleanup: PruneGlobalPhase2 4096
    [11/9/2013 11:37:13 AM 1020] Insufficient RAM for sorting.
    [11/9/2013 11:37:13 AM 1020] Client Backup server failed at d:\wssg_src\whs_pp3\qhs\src\backup\util\sort.cpp(647)

    I also have the same hardware mentioned above.  I am fairly suspicious that adding more physical RAM to the device would actually solve the problem since it should just be allocating virtual memory if needed to satisfy a request, but I acknowledge it could still be an issue.  Has anyone resolved this specific issue, either through some sort of workaround short of deleting all the backups, or by adding more physical RAM?

    Sunday, November 10, 2013 4:30 PM
  • Has anyone successfully resolved this problem without wiping out the backup DB and losing all your backups?  I have precisely the problem described above and get the two stated errors ...

    Mike ... I'm the one who started this thread.  In brief, I have no new solid answers for you.  However, I'm taking a moment to describe how I've decided to proceed.

    Let me preface my comment: I do NOT necessarily think the direction I'm taking is the best for everyone.  But I think my decision might be best for me ... with emphasis on the "might be."

    I've learned from other forums not affiliated with Microsoft that other WHS users have also had this problem. I'm gonna take a wild guess as to why this may be happening to some folks, but not everyone.  Having extraordinarily large files could be the culprit.  In my case, I have one particular video file that's over 40 GBytes in size.  Once that became part of my database, I'm suspecting it may be the cause for certain functions to stumble, like deleting old backups.  Again, though, that's just a guess.

    Detailed instructions are available at these other forums I alluded to show you exactly how to disassemble your HP server and install more memory.  In addition, Olaf, an MVP contributor, mentioned using one of the popular WHS add-ons that can be used to backup your server. I feel that's an important tip especially if some data on your server doesn't also exist elsewhere (meaning your WHS copy is your only copy).  Thank you, Olaf, for mentioning that.

    I'm down to 30% available space and thus far I've been very lucky; I mean, except for an occasional failed backup, my WHS database has remained intact.  While I like many things about WHS, in my UNinformed opinion, the database used to store your backups may be especially sensitive, and thus vulnerable to becoming corrupted or lost more easily than with other backup solutions.  I mean, I've been fearing that if a stroke of bad luck comes my way, I could lose a backup or possibly the entire database, thus the importance of the backups that Olaf recommended.  So, in my opinion, perhaps a simpler, less-vulnerable, and more-resilient backup solution is needed.

    Therefore, I feel that leaves me three options: (1) Add more memory to my server and hope that fixes the problem; (2) Abandon WHS and get several external hard drives to manually image my drives and manage the backups; or (3) Migrate to using a NAS.  In my case I think the Synology DS412+ NAS might fit the bill.  By selecting the correct RAID configuration, at least I'll be able to suffer the loss of one disk drive without losing everything.  Hopefully, by taking this route, I shouldn't have to worry anymore about backing-up my backups.  When my storage needs grow, I think scaling up might be easier, too.  As I said, though, other people's situations may be different, of course, plus some of the notions I've expressed might not be 100% correct.

    I've often wondered what other WHS users are planning to do.  What ever you do, good luck with it.

    Tony M.


    • Edited by TonyM408 Sunday, November 10, 2013 6:01 PM Clarification
    Sunday, November 10, 2013 5:55 PM
  • Adding more memory to the appliance is definitively a good idea, at least if you got one with only 512 MB of RAM.

    In some cases the Mediasmart seems not to work well with 2 GB module, since I experienced crashes of WHS and especially the network driver after a few days of running flooding the attached switch. Downgraded to 1 GB, stable since then.

    From time to time its not a bad idea, to wipe out the entire backup database (and eventually replace the older server disks after done before backing up the clients from the scratch). (If needed, restore volumes from no longer existing machines to a drive attached to a current PC, and back those up later again to the refreshed backup database.)

    The purpose of WHS backup in the eyes of Microsoft was never a long lasting backup, it was always only considered to be a quick way to recover a just failed system or a just lost file. And it was a compromise - pack as much data from the PC into as low space as possible, so redundancy and robustness were not a part of it.

    But this concept has served me well in the past, you have only to know it - and check the reliability of backups from time to time.

    RAID is not a replacement for backup, only a risk reduction. One disaster, affecting the entire server (i.e. overvoltage) can damage all disks at once, as also a controller failure etc. And it also has happened in the past, that while rebuilding the RAID after a disk failure a second disk became faulty.

    Best greetings from Germany
    Olaf

    Monday, November 11, 2013 8:19 AM
    Moderator
  • Thank you for your reply, Olaf.  I don't think anyone can differ with your suggestions and additional information about WHS, RAID and backup strategies.  Your contributions are greatly, greatly appreciated.

    Tony M.

    Monday, November 11, 2013 1:15 PM
  • Thanks to you both for the additional comments and suggestions.  I do agree that WHS V1 is a relatively old and no longer supported technology, but it's been working well for me for many years so I'll probably continue to stick with it. At least until something comes along that piques my interest enough to invest in a new solution (windows server 2012 essentials?) ;-)

    I probably won't install the extra memory on the server given there is no conclusive evidence that it will solve my problem, the addition of 2GB appears to sometimes be problematic, and although my current server is admittedly sometimes slow it works fine for what I use it for.

    Yes, I have used WHS BDBB for several years to backup my data, and that has been very useful.  My last resort plan to resolve this particular problem, short of a real solution that allows me to keep the backups on the server, is to do a fresh backup of all my data to external disk via WHS BDBB as an archive, wipe my entire backup database on the server, and start from scratch.  If I need any data from that backup in the future (hopefully not!) I can save my current set of backups to external disk, restore the archived backup, recover whatever I need to from that, wipe the backup database again, and finally restore back the current backup database from the external drive after I am done.

    Hmm, I really hope I never have to do that!

    Tuesday, November 12, 2013 1:44 AM
  • Hmm, I really hope I never have to do that!

    Isn't that what we all hope if it comes to the restore part of backups? ;-)

    Other than that the concept sounds good enough - especially, since for this old platform no structure changing updates will come up any more. (Of course, if it makes sense depends also from the total size of your backup database - in one of my home servers, which is an additional protection for my company servers, its consuming over 4 TB, so the external disk solution becomes difficult.)

    Best greetings from Germany
    Olaf

    Tuesday, November 12, 2013 7:02 AM
    Moderator
  • Can WHS BDBB backup to a hard drive > 2TB?  I've searched around and it isn't clear to me if that will work.  I'm hoping if the > 2TB drive supports 512e/advanced format then it will work?  I was just about to perform my BDBB backup (to implement the plan outlined above) but as I feared since it has been unable to cleanup backups for some time the backup database will no longer fit on my 2TB backup drive. :-(
    Friday, November 22, 2013 3:41 AM
  • Since it is basically a copy of the folder structure with the physical files, I would assume, it should work. But since I dont have disks of this size at my hands and did not program the tool, its more or less try and error.

    I'm not fully sure, if the combination of WHS and your hardware and its drivers is capable to read and write large GPT formatted disks in the beginning.

    Best greetings from Germany
    Olaf

    Saturday, November 30, 2013 7:14 PM
    Moderator
  • Yes, let me close on this thread by indicating what I did in the hopes that it may help someone else out who runs into this.

    I was able to connect a 3TB hard drive (WD My Book) to my WHS via USB 2.0 interface.  The WD My Book doesn't have an eSata interface and my WHS doesn't support USB 3.0 unfortunately.  Although I considered it, I didn't try to remove the bare 3TB drive from the WD case and plug it into my remaining open WHS drive bay.  Although using USB 2.0 was extremely slow, after approximately 1.5 days WHS BDBB was able to copy the entirety of my 1.94 TB backup DB (encompassing backups from 10 different PCs) to the 3TB drive!  I have now archived this drive away and honestly hope to never use it again - but it's great to know all that data is still there if I should need some old backup some day from a defunct PC!

    I then used WHS BDBB to delete the backup DB so I could start from scratch.  This freed up the nearly 2TB of space.  I then ran discovery.exe on all my clients to reconnect them and re-establish a backup relationship with my WHS and started doing manual backups.  I now have initial backups of my 5 active PCs consuming about 300GB of server space.  It's great to see my WHS with 49% free space and 1.81 TB to work with!  After that I have verified that daily backups are occurring properly, and that the backup "Cleanup Now" functionality successfully completes (and cleaned up some marked for deleted backups.)

    A few random things I've noticed now is that it takes typically two attempts to connect to the WHS Console from my PCs, not sure why it fails the first time.  I installed a couple of plugins (WHS Toolkit 1.1 and Home Server SMART Classic v3.1) to help with troubleshooting this problem so perhaps they are causing it.  Also, the latter plugin is able to provide basic data about the drives but unfortunately it indicates that SMART status is unknown for all of them.  Perhaps that is a limitation of my hardware?

    Anyhow, generally my WHS is working great now.  I accomplished my main goal which was to not lose all of my backup history, but also free up significant storage space and be able to maintain that with backup cleanups so that I have enough headroom to hopefully last several more years now. 

    Thanks for your support Tony and Olaf helping me consider my options.

    Sunday, December 1, 2013 7:40 PM
  • Well, Mike, although I may have had an idea or two, Olaf is the one bringing lots of WHS expertise to the forum.  Thank you, though.

    As for your technique/method, it seems you're saying you freed up space on your server that had previously been used by old backups of your active systems, and by backups of old machines no longer in service.  Is that correct?  I've never used WHS BDBB, but I looked it up to confirm what I thought it was.  Your work proved that was a good idea.

    I have mixed feelings about WHS simply because I don't know enough about the database's inherent ability to resist becoming corrupted.  I know there's a recovery utility, but thank God I've never needed to use it.  So in my case, 90% of the backups on my WHS server represent files on my active client workstations.  The other 10% represents data existing only on the server, so that's why my fingers have been crossed.  Obviously, I should be looking further into the WHS BDBB you mentioned just to see how it might be able to help me.

    My HP WHS sever is 70% full right now.  I don't know whether I can take it all the way up to 100% or not, but I'm not going to push my luck.  Soon I'll do one of two things: Try to replace my smallest drive with a larger drive, or I'm also considering a four-bay NAS unit instead.

    I mentioned my four-bay NAS idea (two drives for data and two drives for mirroring) in an earlier post, but Olaf made some good points: "RAID is not a replacement for backup, only a risk reduction. One disaster, affecting the entire server (i.e. overvoltage) can damage all disks at once, as also a controller failure etc. And it also has happened in the past, that while rebuilding the RAID after a disk failure a second disk became faulty."  I'd have the NAS on an UPS just as my WHS server is, but UPS units fail, too, I realize.  In the end, there's no substitute for have two totally independent copies ... with "independent" meaning on different devices, preferably at different locations.

    Olaf is correct, of course.  I've seen people who very reluctantly begin the practice of backing up their data; for novices, it's a new concept.  However, when it's time to do another backup, because the device they used has insufficient space for multiple backups, they begin their next backup by erasing the only backup they had, and then begin a new backup.  They're not realizing that if their system dies before their new backup is complete, they're sunk.  That's pretty basic, of course, but some folks don't get it.

    Thank you, Mike, and thanks to Olaf, too.





    • Edited by TonyM408 Sunday, December 1, 2013 9:48 PM
    Sunday, December 1, 2013 9:43 PM
  • Hi Tony, let me answer your questions here:

    Yes, I freed up space on the server by deleting all of the old backups. That consisted of backups from both old machines no longer in service and current machines.  I used whs bdbb to do this as it has a button that automates it, but it is effectively a delete of a single folder's contents as documented here:http://www.microsoft.com/downloads/details.aspx?FamilyID=196fe38c-df20-4e19-92ca-6bda7bec3ecb&DisplayLang=en

    This basically reclaimed all my backup space, "fixed" my problems with being unable to clean up those backups, and of course didn't touch the shared folders on the WHS.  After that, as I outlined above, I reconnected my current PCs to the WHS by running discovery.exe and then did full initial backups.  Thus, I now have a backup of all my current machines on my WHS, but of course don't have my old machines backed up on it any more nor do I have any backup history of my current machines before I just wiped and created new ones on my WHS.  However, I do have an old archived copy of all of that data on an external drive if I should ever need it in the future.

    With regards to WHS BDBB, I've used it for several years now and I find it invaluable. I want data and machine backups that are geo-redundant, not just data spread across multiple drives or even different machines.  For me, that means that every 6 months I hook up an external drive to my WHS and use the built-in utility to back up all the shared folders to the drive.  In addition, at that 6 month interval I also use WHS BDBB to back up my entire backup database to a separate external drive.  My WHS is at home so I store these two external backup drives at work.  If something horrible happens like my WHS gets stolen, fried, burned up in a fire, etc. I have those backups at work.  This works for me because I don't have so much data (at this point I have approximately 1 TB of data "live" backed up on my WHS - doesn't include my single 1.94TB snapshot of old archived data) that would make this process impractical.  It fits comfortably on two external drives.  Your mileage may vary and everyone needs to decide how much effort they want to expend to manage their backup data, but this is what works for me and provides a level of confidence and redundancy that I am personally comfortable with.

    If you haven't already, I strongly recommend installing and using WHS BDBB on a regular basis.

    Best regards.

    Saturday, December 7, 2013 10:08 PM