none
File DELETE PENDING problem

    Question

  • I don’t know where the best location to ask for help with this problem is, and while I would like to see a fix come out of this, at this least this post will document the problem for anyone else who encounters a similar one.  I have some questions and requests for help near the end of the post for anyone who would like to help.

    We have a large c# solution in Visual Studio 2012 Update 4 on Win7 made up of many different Class Library projects and several Console and Windows Applications.

    Compiling many times a day, sometimes one of the projects fails with an Access Denied error on one of the projects PDB file (around once or twice a day).  The project affected is usually different each time. The only fix for this error has been to reboot the computer and then continue as usual until next time.  We have several developers working on this code base, but only 2 so far are seeing this issue.

    When investigating the problem, we have found that despite the error being reported on the projects PDB file, the cause of the problem seems to be a .original file.  Assuming the affected project is a Class Library project, the original file seems to be a copy of the dll output named <Project>.dll.original.  The file can't be opened or deleted by an administrator, and taking ownership also fails.

    Using SysInternals Process Explorer, we can see that no processes have a file handle open on the file.

    Using SysInternals Process Monitor (a capture reproduce below), we can see that the failing operations are resulting in a DELETE PENDING result for any attempt to access the file after the process that creates it (ccrewrite), deletes it (or at least attempts to).  Normally this delete works, in these cases it seems to leave the file in limbo where it can't be accessed or deleted, and nothing has it open.

    After a reboot, the issue is fixed and we have full control and access to the file.

    The systems are running McAfee virus scanning software, and Windows Search is also indexing the file locations, but again, from Process Monitor, we can see these processes do not appear to be interfering significantly in these instances (open file handles are being closed before the delete is initiated).

    Other similar problems (found via Google) related to specific code rather than Visual Studio tools do not report the same issue.  When the file is DELETE PENDING, it is usually deleted when all file handles are closed, it is just that there may be a delay.  In this case it never happens.  After the reboot, the files are not deleted, but are also no longer PENDING DELETE.

    After looking at http://www.sevenforums.com/tutorials/17854-index-locations-add-remove.html I identified what was putting the .original file into the never ending DELETE PENDING state.  The Windows Indexing Service sometimes fails to index the file with a result of OPLOCK NOT GRANTED in Process Monitor.  The file is being created in a location that is configured to be indexed.  So far, I have observed that when this happens, the following delete will put the file into the never ending DELETE PENDING state.

    I have also created a little c++ program that makes the problem happen (on our systems).

    #include <SDKDDKVer.h>

    #include <windows.h>

    #include <stdio.h>

    #include <tchar.h>

    int _tmain(int argc, _TCHAR* argv[])

    {

      if (argc < 2)

      {

        _tprintf_s(_T("\"%s\" <filename> [-/+<extension for rename>]\n"), argv[0]);

        return 0;

      }

      _TCHAR *sFileName = argv[1];

      _TCHAR sNewfileName[255];

      _tcscpy_s(sNewfileName, sFileName);

      _TCHAR *sExtension = _T(".original");

      bool bAddExtension = true;

      if (argc >= 3)

      {

        sExtension = argv[2];

        switch(sExtension[0])

        {

        case '-':

        {

          _TCHAR *pEndOfName = _tcsstr(sNewfileName, _T("."));

          *pEndOfName = 0;

          ++sExtension;

          break;

        }

        case '\0':

          printf_s("Invalid extension\n");

          return 0;

        default:

        case '+':

          ++sExtension;

          break;

        }

      }

      _tcscat_s(sNewfileName, sExtension);

      _tprintf_s(_T("Original filename: %s\nRename filename:   %s\n"), sFileName, sNewfileName);

      for (int i=0; i < 100; ++i)

      {

        {

          HANDLE hHandle =  CreateFile(sFileName, GENERIC_WRITE, 0, NULL, CREATE_ALWAYS, FILE_ATTRIBUTE_NORMAL, NULL);

          char data[] = "test";

          DWORD dwDataWritten;

          WriteFile(hHandle, data, _countof(data), &dwDataWritten, NULL);

          CloseHandle(hHandle);

        }

        if (MoveFile(sFileName, sNewfileName) == FALSE)

        {

          printf_s("Error found on try %d!!!\nIt probably occurred on the last try\n", i);

          return 0;

        }

        DeleteFile(sNewfileName);

      }

      printf_s("Problem not detected\n");

      return 0;

    }

    If I pass in a file name that ends in .txt instead of .dll, McAfee no longer looks at the file, and I also no longer get the problem.

    Similar for using an indexed location, if the file location is changed to one that is not indexed, the problem no longer occurs.

    It seems to be a joint issue between McAfee and the Windows Indexing Service.

    Hardware wise, this is being reproduced on systems with 4 cores and SSDs.  I have also tried this on a system with an SSD and a normal HDD.  When the program above is run on this system using the HDD location, the problem is not reproduced, but on the SSD location, it is.

    I have tried to reproduce this on a VM using Windows Defender, but I have not managed to reproduce the issue, and I just don’t know if it is a case of not enough performance on the platform or only McAfee is part of, or the cause, or something else is part of the problem.

    We are using McAfee VirusScan Enterprise 8.8.06000.

    Please try the reproduction program above with the filename parameter being in an indexed location and report your results (and setup).

    Example: reproduction.exe c:\indexedDir\test.dll

    Given Process Explorer is not able to find any open file handles on the file, and the Process Monitor logging to me shows everything is in order, is it possible this is an OS problem?

    Too me it looks like it, it is just not straight forward to set off.  A file in the DELETE PENDING state should be removed when there are no longer any file handles open for that file.

    Has anyone else seen a problem like this?

    While this is not a Visual Studio problem, it initially appears to be one from the symptoms.  I think a different file access pattern may mitigate the problem, because using my reproduction, if the first file is not a dll, but the second file is, there is still no problem.  This would mean a change to ccrewrite for a problem that is probably better solved in the OS or the virus scanner.

    If this is an OS problem what is going to be the best way to raise and address this problem?

    We already have a workaround (don’t use an indexed location), but if this is really an OS problem, it should be fixed.

    Following are what I think are the interesting parts of the event capture from Process Monitor for the original problem (with the important parts in bold).  I have retained all the events to do with the .original file from it not existing to it being in a DELETE PENDING state.  I have only kept events that I think are relevant for the source file (showing it has been moved/renamed).

    "Time of Day","Process Name","PID","Operation","Path","Result","Detail"
    "3:27:26.1091782 PM","ccrewrite.exe","13460","CreateFile","C:\test\tests\obj\Debug\tests.dll.original","NAME NOT FOUND","Desired Access: Read Attributes, Dis
    "3:27:26.1097588 PM","ccrewrite.exe","13460","CreateFile","C:\test\tests\obj\Debug\tests.dll.original","NAME NOT FOUND","Desired Access: Read Attributes, Delete, Dis
    "3:27:26.1120745 PM","ccrewrite.exe","13460","CreateFile","C:\test\tests\obj\Debug\tests.dll.original","NAME NOT FOUND","Desired Access: Read Attributes, Dis
    "3:27:26.1127774 PM","ccrewrite.exe","13460","CreateFile","C:\test\tests\obj\Debug\tests.dll","SUCCESS","Desired Access: Read Attributes, Delete, Synchronize, Dis
    "3:27:26.1128575 PM","ccrewrite.exe","13460","QueryAttributeTagFile","C:\test\tests\obj\Debug\tests.dll","SUCCESS","Attributes: A, ReparseTag: 0x0"
    "3:27:26.1128724 PM","ccrewrite.exe","13460","QueryBasicInformationFile","C:\test\tests\obj\Debug\tests.dll","SUCCESS","CreationTime: 8/03/2016 3:13:13 PM, LastAccessTime: 16/03/2016 12:33:12 PM, LastWriteTime: 16/03/2016 3:27:23 PM, ChangeTime: 16/03/2016 3:27:23 PM, FileAttributes: A"
    "3:27:26.1130173 PM","ccrewrite.exe","13460","SetRenameInformationFile","C:\test\tests\obj\Debug\tests.dll","SUCCESS","ReplaceIfExists: False, FileName: C:\test\tests\obj\Debug\tests.dll.original"
    "3:27:26.1138891 PM","ccrewrite.exe","13460","CloseFile","C:\test\tests\obj\Debug\tests.dll.original","SUCCESS",""
    "3:27:26.1140604 PM","mcshield.exe","3760","CreateFile","C:\test\tests\obj\Debug\tests.dll.original","SUCCESS","Desired Access: Read Attributes, Dis
    "3:27:26.1141167 PM","mcshield.exe","3760","QueryBasicInformationFile","C:\test\tests\obj\Debug\tests.dll.original","SUCCESS","CreationTime: 8/03/2016 3:13:13 PM, LastAccessTime: 16/03/2016 12:33:12 PM, LastWriteTime: 16/03/2016 3:27:23 PM, ChangeTime: 16/03/2016 3:27:26 PM, FileAttributes: A"
    "3:27:26.1141413 PM","mcshield.exe","3760","CloseFile","C:\test\tests\obj\Debug\tests.dll.original","SUCCESS",""
    "3:27:26.1146495 PM","mcshield.exe","3760","QueryDirectory","C:\test\tests\obj\Debug\tests.dll.original","SUCCESS","Filter:
    tests.dll.original, 1: tests.dll.original "
    "3:27:26.1147032 PM","SearchProtocolHost.exe","10188","CreateFile","C:\test\tests\obj\Debug\tests.dll.original","SUCCESS","Desired Access: Generic Read, Dis
    "3:27:26.1148222 PM","devenv.exe","8344","CreateFile","C:\test\tests\obj\Debug\tests.dll","NAME NOT FOUND","Desired Access: Read Attributes, Dis
    "3:27:26.1149586 PM","SearchProtocolHost.exe","10188","FileSystemControl","C:\test\tests\obj\Debug\tests.dll.original","OPLOCK
    NOT GRANTED","Control: FSCTL_REQUEST_FILTER_OPLOCK"
    "3:27:26.1149888 PM","SearchProtocolHost.exe","10188","CloseFile","C:\test\tests\obj\Debug\tests.dll.original","SUCCESS",""
    "3:27:26.1154113 PM","devenv.exe","5096","CreateFile","C:\test\tests\obj\Debug\tests.dll","NAME NOT FOUND","Desired Access: Read Attributes, Dis
    "3:27:26.1168301 PM","devenv.exe","5096","CreateFile","C:\test\tests\obj\Debug\tests.dll.original","SUCCESS","Desired Access: Read Attributes, Dis
    "3:27:26.1168773 PM","devenv.exe","5096","QueryBasicInformationFile","C:\test\tests\obj\Debug\tests.dll.original","SUCCESS","CreationTime: 8/03/2016 3:13:13 PM, LastAccessTime: 16/03/2016 12:33:12 PM, LastWriteTime: 16/03/2016 3:27:23 PM, ChangeTime: 16/03/2016 3:27:26 PM, FileAttributes: A"
    "3:27:26.1169040 PM","devenv.exe","5096","CloseFile","C:\test\tests\obj\Debug\tests.dll.original","SUCCESS",""
    "3:27:26.1185329 PM","ccrewrite.exe","13460","CreateFile","C:\test\tests\obj\Debug\tests.dll","NAME NOT FOUND","Desired Access: Read Attributes, Dis
    "3:27:26.1185449 PM","devenv.exe","8344","CreateFile","C:\test\tests\obj\Debug\tests.dll.original","SUCCESS","Desired Access: Read Attributes, Dis
    "3:27:26.1186177 PM","devenv.exe","8344","QueryBasicInformationFile","C:\test\tests\obj\Debug\tests.dll.original","SUCCESS","CreationTime: 8/03/2016 3:13:13 PM, LastAccessTime: 16/03/2016 12:33:12 PM, LastWriteTime: 16/03/2016 3:27:23 PM, ChangeTime: 16/03/2016 3:27:26 PM, FileAttributes: A"
    "3:27:26.1186558 PM","devenv.exe","8344","CloseFile","C:\test\tests\obj\Debug\tests.dll.original","SUCCESS",""
    "3:27:26.1201736 PM","ccrewrite.exe","13460","CreateFile","C:\test\tests\obj\Debug\tests.dll","NAME NOT FOUND","Desired Access: Read Attributes, Dis
    "3:27:26.1212129 PM","ccrewrite.exe","13460","CreateFile","C:\test\tests\obj\Debug\tests.dll","NAME NOT FOUND","Desired Access: Read Attributes, Dis
    "3:27:26.1669430 PM","ccrewrite.exe","13460","CreateFile","C:\test\tests\obj\Debug\tests.dll.original","SUCCESS","Desired Access: Read Attributes, Dis
    "3:27:26.1669732 PM","ccrewrite.exe","13460","QueryBasicInformationFile","C:\test\tests\obj\Debug\tests.dll.original","SUCCESS","CreationTime: 8/03/2016 3:13:13 PM, LastAccessTime: 16/03/2016 12:33:12 PM, LastWriteTime: 16/03/2016 3:27:23 PM, ChangeTime: 16/03/2016 3:27:26 PM, FileAttributes: A"
    "3:27:26.1669803 PM","ccrewrite.exe","13460","CloseFile","C:\test\tests\obj\Debug\tests.dll.original","SUCCESS",""
    "3:27:26.1674406 PM","ccrewrite.exe","13460","CreateFile","C:\test\tests\obj\Debug\tests.dll.original","SUCCESS","Desired Access: Read Attributes, Delete, Dis
    "3:27:26.1675861 PM","ccrewrite.exe","13460","QueryAttributeTagFile","C:\test\tests\obj\Debug\tests.dll.original","SUCCESS","Attributes: A, ReparseTag: 0x0"
    "3:27:26.1675990 PM","ccrewrite.exe","13460","SetDispositionInformationFile","C:\test\tests\obj\Debug\tests.dll.original","SUCCESS","Delete: True"
    "3:27:26.1677019 PM","ccrewrite.exe","13460","CloseFile","C:\test\tests\obj\Debug\tests.dll.original","SUCCESS",""
    "3:27:26.3194809 PM","SearchProtocolHost.exe","10188","CreateFile","C:\test\tests\obj\Debug\tests.dll.original","DELETE PENDING","Desired Access: Generic Read, Dis, Options: Open Reparse Point, Open Requiring Oplock, Attributes: N, ShareMode: Read, Write, Delete, AllocationSize: n/a"
    "3:27:26.3199501 PM","SearchProtocolHost.exe","10188","CreateFile","C:\test\tests\obj\Debug\tests.dll.original","DELETE PENDING","Desired Access: Read EA, Read Attributes, Read Control, Dis, Options: Open Reparse Point, Open Requiring Oplock, Attributes: N, ShareMode: Read, Write, Delete, AllocationSize: n/a"

    Thank you,
    Warren.


    Friday, March 18, 2016 1:34 AM

Answers

All replies

  • Hi Warren,

    Actually this forum is to discuss the VS IDE usage.

    I try to test the sample provided in my side, sorry for that I also couldn't repro this issue. The project runs normally, and no other new files were created automatically in my side.

    >>We have several developers working on this code base, but only 2 so far are seeing this issue.

    Since the same code just has this issue in two machines. I think the real issue would be not the project itself. The VS Environment or the Windows Environment would be the possible reason.

    Of course, to make sure that it is not the VS setup issue, you could repair your VS, please also install the latest VS2012 update 5, test it again.

    Or you could use http://aka.ms/vscollect  to gather the VS and .Net Frameowrk installation logs. After using it, you will find vslogs.cab from %temp% folder. Please upload the file to Onedrive and share the link here.

    But if no setup error, I think it would be related to the specific windows Environment.

    >>It seems to be a joint issue between McAfee and the Windows Indexing Service.

    Could you remove/uninstall the specific virus software? And then re-test it again.

    To make sure that whether it is related to any add-ins or third party tools, maybe you could run your VS in safe mode, check the result.

    Devenv.exe /SafeMode

    But as you said that if it is not the VS issue, I mean that maybe it is the OS issue, I have no good suggestion, maybe you could re-install a new clean OS, check it again. At least, it would be safe for project source controls if other machines have no this issue. 

    Best Regards,

    Jack


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Monday, March 21, 2016 9:33 AM
  • Hi Jack,

    Thank you for the response and I apologise for being slow getting back.  This issue is quickly falling in my list of priorities because we do have a functional workaround, but I would still like to find a real fix because I don’t think anything at the application level is being done wrong.

    Actually this forum is to discuss the VS IDE usage.

    Sorry for posting to the wrong location, but I was looking for somewhere that might be relevant (for someone not knowing the actual cause of the problem).

    I try to test the sample provided in my side, sorry for that I also couldn't repro this issue. The project runs normally, and no other new files were created automatically in my side.

    Thank you for running the sample, but unless your platform is sufficient, from what I have seen, you will not see the problem.  I have done some additional testing since posting the initial post, and on one PC I have access to, there are two SSD installed.  One is decent (http://ssd.userbenchmark.com/OCZ-Vertex-3-120GB/Rating/1927), the other is slow (http://ssd.userbenchmark.com/SpeedTest/10978/SAMSUNG-MMCRE64G5MXP-0VB).  Running the sample using the decent OCZ drive reproduces the problem, but using the slow Samsung does not.  I have also tried running the sample program without McAfee and using Windows Defender instead, but only in VM environments.  Given the performance requirements to see the problem in the first place, I don’t know at this stage if it is a lack of performance that means I do not see the issue, or if it is the lack of McAfee.

    >>We have several developers working on this code base, but only 2 so far are seeing this issue.

    Since the same code just has this issue in two machines. I think the real issue would be not the project itself. The VS Environment or the Windows Environment would be the possible reason.

    Of course, to make sure that it is not the VS setup issue, you could repair your VS, please also install the latest VS2012 update 5, test it again.

    Or you could use http://aka.ms/vscollect  to gather the VS and .Net Frameowrk installation logs. After using it, you will find vslogs.cab from %temp% folder. Please upload the file to Onedrive and share the link here.

    But if no setup error, I think it would be related to the specific windows Environment.

    I am already quite certain it is nothing to do with VS.  It is just that when using VS for c# development, and you are also using dotnet CodeContracts, you may get this issue.  The CodeContracts program ccrewrite could be changed to work around the problem, but that is all it would be, a work around.

    The easiest work around it to not compile these projects in an indexed location, but it takes a lot of head scratching to figure that out.

    >>It seems to be a joint issue between McAfee and the Windows Indexing Service.

    Could you remove/uninstall the specific virus software? And then re-test it again.

    In regards to the virus software, although in the environment where I work I can’t remove the virus software, I have been able to make the virus software ignore the file by change the extension from .dll to .txt.  When I do this, I do not reproduce the problem with my reproduction program.  I do want to collect some data about any other virus scanners that do contribute to the problem, but the only way of doing this for sure is to find a system with McAfee where the problem can be produced, then uninstall McAfree and try various other virus scanners.

    To make sure that whether it is related to any add-ins or third party tools, maybe you could run your VS in safe mode, check the result.

    Devenv.exe /SafeMode

    But as you said that if it is not the VS issue, I mean that maybe it is the OS issue, I have no good suggestion, maybe you could re-install a new clean OS, check it again. At least, it would be safe for project source controls if other machines have no this issue. 

    I think I am already past that.  The only parts of the system that put the files into a DELETE PENDING state are ccrewrite.exe, SearchProtocolHost.exe and mcshield.exe.  When compiling, devenv.exe is also involved, but using my reproduction program devenv is not, and the reproduction replaces ccrewrite.

    At this stage, I think the two avenues of approach are going to be McAfee and/or the Microsoft OS group.  While I can see a finger pointing stalemate as a possibility, I also think it should be possible to modify the reproduction so that McAfee can be removed from the picture and it becomes solely an OS issue.

    So while I think the issue in this forum is closed, I don’t think it is resolved (and mainly because this not really the right place).

    If anyone else can help with more data about systems where the problem can be seen, particularly in relation to virus scanners or VM environments, that would be useful information.

    Thank you,
    Warren.


    Wednesday, March 30, 2016 12:03 AM
  • Hi Warren,

    Thanks for your friendly response.

    >>So while I think the issue in this forum is closed, I don’t think it is resolved (and mainly because this not really the right place).

    Since the main issue is not the VS IDE usage issue, I am moving your question to the moderator forum ("Where is the forum for..?"). The owner of the forum will direct you to a right forum.

    >>If anyone else can help with more data about systems where the problem can be seen, particularly in relation to virus scanners or VM environments, that would be useful information.

    If it is related to the virus scanners, you would post this issue to the software's official site where you download it.

    But if it is the windows issue, maybe this forum would be better:

    http://answers.microsoft.com/en-us/windows

    Anyway, I move it to the moderator forum. Thanks for your understanding.

    Sincerely,

    Jack


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Wednesday, March 30, 2016 11:46 AM
  • If the issues are with the operating system I'd try over here.

    http://answers.microsoft.com/en-us/windows/forum/windows_7

    https://social.technet.microsoft.com/Forums/windows/en-US/home?forum=w7itprogeneral

     

     

     


    Regards, Dave Patrick ....
    Microsoft Certified Professional
    Microsoft MVP [Windows Server]

    Disclaimer: This posting is provided "AS IS" with no warranties or guarantees , and confers no rights.

    Wednesday, March 30, 2016 1:20 PM
    Moderator
  • Hi Dave,

    thanks for the pointer.  I have posted simpler description of this problem to the Windows 7 TechNet forum (but it refers back to this post too).

    Edit:
    See https://social.technet.microsoft.com/Forums/windows/en-US/58e5c670-f024-44ff-9919-36c44ab11d9c/file-delete-pending-problem?forum=w7itprogeneral#58e5c670-f024-44ff-9919-36c44ab11d9c.

    So in this thread, as to where to ask about this problem, I will mark it as answered.

    Thanks,
    Warren.


    Thursday, March 31, 2016 6:22 AM
  • Sounds good, you're welcome.

     

     


    Regards, Dave Patrick ....
    Microsoft Certified Professional
    Microsoft MVP [Windows Server]

    Disclaimer: This posting is provided "AS IS" with no warranties or guarantees , and confers no rights.

    Thursday, March 31, 2016 8:15 AM
    Moderator