locked
Tombstones pointing to a non-storage pool drive RRS feed

  • Question

  •  This is weird. First a bit of background:

    One of my storage pool drives started failing (delayed write failures on $mft and $bitmap). After some tests (chkdsk, replace sata cables, connect to drive to different port, reboots) it kept happening. I may have swapped a couple drive-to-port relationships - I had to reset the boot priority in the BIOS at one point. Anyway, I ended up removing it from the pool. There was plenty of free space on other drives, no files would be lost. The failing drive is 500GB with about 92GB of backups, and about 340GB of mostly duplicated files. I expected it to take quite a while to move the backups. I assumed that migrator would catch up the duplication after the removal.

    I was away briefly, maybe 10 minutes at most, and when I came back the remove drive wizard was done. Wow I thought, pretty quick. I shutdown to remove the drive.

    Next day, heaps of file conflicts on about 8 of the shares, including all of the shares with duplication turned on. The conflict error is "The system cannot find the drive specified". Sure enough when I tried accessing the files, they really weren't there. Only tombstones. I start thinking it's an appropriate name ;-)

    Ok I figure I can probably recover a lot of the data from the removed drive. I put it back into the server and add it as a backup drive with existing data preserved.

    I needed an easy way to find the problem files. The duplication info addin was helpful. It shows lots of files are not on any of the drives. I check some, but they are there, the duplication info addin just can't see them.

    Ok so the tombstones are pointing to the files now. But where are they?

    I figured it out - the tombstones are still pointing to the shadows on the 500GB drive, which I had removed from the pool, and subsequently re-added as a backup drive. Arrgh!

    Worse still, I can now go into a share, create a folder, copy some files in, and guess where the shadow files end up... on the backup drive. More Arrgh!

    I also removed the drive again, but left it connected, and WHS still access and creates shadows on it :-/

    I have the 500GB drive connected and sitting outside the case at the moment, it seems ok. I suspect its problem was too much vibration adjacent to other drives in the case. As far as I can tell, (according to chkdsk), I still have all of my files on it.

    I suspect that for some reason the remove drive wizard has not invalidated the shadow pointers in the tombstones when I removed the drive from the storage pool.

    Now the questions:

    Is it only the references in the tombstones that lead to WHS accessing the 500GB drive, or is there something else in WHS's brain that makes it think that the drive is part of the pool?

    Is there a tool that can fix the messed up tombstones?

    If I wait a while, will migrator fix the tombstones?

    I want to put the 500GB drive in another PC, and then copy the folders under /DE into the shares in order to put the data back, but I don't want to do that while WHS thinks this drive is still part of the storage pool. Can I make WHS stop thinking that? Maybe after I save all of the data somewhere, put it back in the pool, and take it out again?

    Also, can I get the 92GB backup data back into the back up database somehow? (I have turned off backups for now. Not sure what impact more backup data might have on this problem.)


    Any guidance to clear this up would be much appreciated.

    Paul.
    Thursday, September 11, 2008 3:13 PM

Answers

  • Hi Paul,

    This definitely shouldn't be happening. I think you should file a bug report, instructions overher: Sticky: File Conflict Error Messages and PP1. Please do this before recovering.

    To recover I would suggest you wait for response to your bug report. If you can't wait:

    Temporary disable all backups (Home Computer Backup, pages 20 - 23), do not copy any data to your WHS

    1. Copy the complete contents of the (hidden) DE folder on the current 500 GB backup drive to another folder.
    2. Then disconnect the 500 GB backup drive.
    3. From the location to which you copied the data in 2
      1. Copy the content from each of the shares folder back to it's original location on the server. Copy to these files / folders using the links to the shares on a client or use the shortcut to the shares folders on your server desktop. This should normally fix most of the tombstones
      2. Backup files should be copied from the folder {00008086-058D-4C89-AB57-A7F909A47AB4} into the folder D:\folders\{00008086-058D-4C89-AB57-A7F909A47AB4}. This has to done from the server desktop (or RDP connection to the server desktop)
    4. This procedure should fix most problems. If you have any problems left you can use the utility from user catsaver to clean them up. If you have all the files in the shares duplicated and don't mind loosing your client backups you can also skip steps 1 and 3.
    • Marked as answer by paulyb Sunday, September 14, 2008 5:11 AM
    Thursday, September 11, 2008 3:47 PM
    Moderator

All replies

  • Hi Paul,

    This definitely shouldn't be happening. I think you should file a bug report, instructions overher: Sticky: File Conflict Error Messages and PP1. Please do this before recovering.

    To recover I would suggest you wait for response to your bug report. If you can't wait:

    Temporary disable all backups (Home Computer Backup, pages 20 - 23), do not copy any data to your WHS

    1. Copy the complete contents of the (hidden) DE folder on the current 500 GB backup drive to another folder.
    2. Then disconnect the 500 GB backup drive.
    3. From the location to which you copied the data in 2
      1. Copy the content from each of the shares folder back to it's original location on the server. Copy to these files / folders using the links to the shares on a client or use the shortcut to the shares folders on your server desktop. This should normally fix most of the tombstones
      2. Backup files should be copied from the folder {00008086-058D-4C89-AB57-A7F909A47AB4} into the folder D:\folders\{00008086-058D-4C89-AB57-A7F909A47AB4}. This has to done from the server desktop (or RDP connection to the server desktop)
    4. This procedure should fix most problems. If you have any problems left you can use the utility from user catsaver to clean them up. If you have all the files in the shares duplicated and don't mind loosing your client backups you can also skip steps 1 and 3.
    • Marked as answer by paulyb Sunday, September 14, 2008 5:11 AM
    Thursday, September 11, 2008 3:47 PM
    Moderator
  • Hi Paul,

    Can you please let us know what you have actually done, and what results you got?


    Sunday, September 14, 2008 5:33 AM
    Moderator
  • Hi brubber,

    I experimented with your recovery plan, but it needs modification. If I try copying anything to the shares while that disk is online in the server (as a backup or an unmanaged disk) some of the data will end up on it. DE still sees it as part of the storage pool.

    I don't think it wise to copy directly into the filesystem to bypass DE, so I plan to install the disk in a client PC, and copy the data from there, as per your recovery plan. For the backups I will copy that 92GB to a share, then kvm in and copy from the share to D:\folders\{00008086-058D-4C89-AB57-A7F909A47AB4}.

    I'm pretty confident that will clean up all the tombstones, barring any genuine data loss. Hopefully the backup db will then be ok too.

    (Powering up the server without the drive causes the backup service to fail and a health notification warning that the backup db needs repair. Not unexpected with 92GB of backup db absent...)

    Cheers.




    Sunday, September 14, 2008 8:15 AM
  • An update. Copying a missing shadow file from the disk (installed in a client PC) over itself in the shares is not straightforward. DE follows the tombstone pointers to the now absent drive, and the client gets this:

    Error 0x8007048F: The device is not connected.

    However, deleting the file from the share works, then the shadow can be copied in.

    This is going to be a very tedious recovery. I might put together a script to do all this work.
    Sunday, September 14, 2008 9:19 AM
  •  Hi Paul,

    You might try the utility I mentiond under 4. in my first reply to find and cleanup the abberant tombstones. Not 100% sure if it will work though since this one was designed to cleanup tombstones with all shadows missing. Just give it a try. It's pretty save since it's a two step program, after finding the abberant shadows it creates a batch file that you can review before executing it. 

    You are right the 500GB drive should be disconnected, mentioned this in point 2. in my first reply.

    I think your current strategy, mounting the disk in a client and then copying (after deleting the tombstones) to the shares should do the trick for the files in the shares. Only for the backup files this will not work. You can either share the backup folder (3.2. in my first reply) and then copy the files in there, or first copy the files to some other share and then from the server desktop move them to the correct location.

    Good Luck!
    Sunday, September 14, 2008 11:27 AM
    Moderator