none
Diagnosing NTFS slowdowns when writing beyond EOF

    Question

  • On some of my systems, writing beyond the EOF results in significant slowdowns (10x), whereas some other systems are unaffected. I am trying to find out what causes this behavior (using procmon, perfmon), but am having a hard time doing so. The NTFS settings I have found so far in perfmon with a default system diagnosis seem to be equal. How would I go about finding out what the difference in configuration is between these systems?

    So far, I know that it only affects NTFS filesystems, and not FAT32. It does not matter what the type of the drive is, e.g. SSD drive, usb stick, or even a RAM disk. If a system is affected, all NTFS partitions perform worse for beyond-EOF writes. It is probably some simple setting that is different, but I have not found it yet.

    (I would include my test results and source code of test programs, but there seems to be no way to let these be collapsed by default. I am afraid they would therefore only clutter this post/topic, but if there is a way of sharing them properly I will happily do so!)

    • Moved by Just KarlModerator Tuesday, March 03, 2015 2:56 PM Looking for the proper forum.
    Friday, February 27, 2015 11:49 PM

Answers

  • After more testing, and suggestions/help of friends, I have found out what causes the slowdowns. It turns out that the culprit is the real-time protection of Microsoft Security Essentials (or the equivalent)!

    Unfortunately I did not look more into real-time protection being the culprit sooner, because my work laptop has "System Center Endpoint Protection". The sysadmins included a filetype exclusion after we noticed slowdowns, which did not help. As such, I figured that the real-time protection could not be the cause. However, as it turns out, this filetype exclusion was not entered correctly; the entry's extension reads something like ".sdt, .sdf, ...". Trying the same on my desktop with Microsoft Security Essentials, I was allowed to only enter a single extension at a time, which then reads "sdt" (note the lack of the dot as well)

    As happy as I am to have found the cause (and workaround/solution), it still mystifies me that the real-time protection chockes on this type of writing behavior. To summarize, slowdowns are experienced when (1) seeking+writing beyond the EOF on (2) files that already existed before you opened them (3) on NTFS file systems. If any of these three factors is not satisfied, no slowdowns are experienced.

    I will mark this post as the answer, as the diagnosing part has been completed (using a.o. procmon and fltmc). I will probably make a another small post in http://answers.microsoft.com/en-us/protect/forum/mse to report the real-time protection's behavior in a more suitable place, and ask for some explanation on why it occurs (out of sheer curiosity).

    (continued in http://answers.microsoft.com/en-us/protect/forum/mse-protect_scanning/real-time-protection-causes-slowdowns-when/35bb4019-6d97-4f4b-be49-c96a0fbf215e?tm=1425230837333 )





    • Marked as answer by dj_tjerk Sunday, March 01, 2015 2:44 PM
    • Edited by dj_tjerk Sunday, March 01, 2015 5:29 PM adding link to MSE forum post
    Sunday, March 01, 2015 2:44 PM

All replies

  • Setup:
    Both of my systems are running Windows 7.
    Other systems that were tested on were either Windows 7 or Windows 8.1. Affected systems ~50% (of Windows 7, I am not sure about Windows 8).

    I have tested on systems with and without virus scanners and the like, but this does not seem to matter. I have tested on various drivers (usb sticks/ram disks), some of which were shared and some were not. This also does not influence the results.

    Diagnosis:
    EOF+1 write performance testing (my results in comments): https://gist.github.com/PietjeBell88/979981f9dff1f917df96

    As this issue may or may not be related to the writing of sparse files, I also tested that analogously to a blog post here on 

    Sparse file write testing (my results in comments): https://gist.github.com/PietjeBell88/e65922968df0286de51c

    As my laptop (the affected system) is actually faster with sparse files than my desktop, my write-beyond-EOF issue seems unrelated.





    • Edited by dj_tjerk Saturday, February 28, 2015 4:52 PM
    Saturday, February 28, 2015 11:09 AM
  • After more testing, and suggestions/help of friends, I have found out what causes the slowdowns. It turns out that the culprit is the real-time protection of Microsoft Security Essentials (or the equivalent)!

    Unfortunately I did not look more into real-time protection being the culprit sooner, because my work laptop has "System Center Endpoint Protection". The sysadmins included a filetype exclusion after we noticed slowdowns, which did not help. As such, I figured that the real-time protection could not be the cause. However, as it turns out, this filetype exclusion was not entered correctly; the entry's extension reads something like ".sdt, .sdf, ...". Trying the same on my desktop with Microsoft Security Essentials, I was allowed to only enter a single extension at a time, which then reads "sdt" (note the lack of the dot as well)

    As happy as I am to have found the cause (and workaround/solution), it still mystifies me that the real-time protection chockes on this type of writing behavior. To summarize, slowdowns are experienced when (1) seeking+writing beyond the EOF on (2) files that already existed before you opened them (3) on NTFS file systems. If any of these three factors is not satisfied, no slowdowns are experienced.

    I will mark this post as the answer, as the diagnosing part has been completed (using a.o. procmon and fltmc). I will probably make a another small post in http://answers.microsoft.com/en-us/protect/forum/mse to report the real-time protection's behavior in a more suitable place, and ask for some explanation on why it occurs (out of sheer curiosity).

    (continued in http://answers.microsoft.com/en-us/protect/forum/mse-protect_scanning/real-time-protection-causes-slowdowns-when/35bb4019-6d97-4f4b-be49-c96a0fbf215e?tm=1425230837333 )





    • Marked as answer by dj_tjerk Sunday, March 01, 2015 2:44 PM
    • Edited by dj_tjerk Sunday, March 01, 2015 5:29 PM adding link to MSE forum post
    Sunday, March 01, 2015 2:44 PM
  • Related references may include:


    ~Robear Dyer (PA Bear) MS MVP-Windows Client since 2002 Disclaimer: MS MVPs neither represent nor work for Microsoft

    Tuesday, March 03, 2015 12:38 AM
  • Hello,

    The Windows Desktop Perfmon and Diagnostic tools forum is to discuss performance monitor (perfmon), resource monitor (resmon), and task manager, focusing on HOW-TO, Errors/Problems, and usage scenarios.

    As the question is off topic here, I am moving it to the Where is the Forum... forum.

    Karl


    When you see answers and helpful posts, please click Vote As Helpful, Propose As Answer, and/or Mark As Answer.
    My Blog: Unlock PowerShell
    My Book: Windows PowerShell 2.0 Bible
    My E-mail: -join ('6F6C646B61726C406F75746C6F6F6B2E636F6D'-split'(?<=\G.{2})'|%{if($_){[char][int]"0x$_"}})

    Tuesday, March 03, 2015 2:54 PM
    Moderator
  • Hello,

    By the way, you were correct to continue this in the Virus and Malware forum on Microsoft Community.

    As the Microsoft Community is on a different platform, we cannot move the question for you.

    Once there, click on Participate near the top of the screen, and select 'Ask a Question' or 'Start a Discussion':

    Karl


    When you see answers and helpful posts, please click Vote As Helpful, Propose As Answer, and/or Mark As Answer.
    My Blog: Unlock PowerShell
    My Book: Windows PowerShell 2.0 Bible
    My E-mail: -join ('6F6C646B61726C406F75746C6F6F6B2E636F6D'-split'(?<=\G.{2})'|%{if($_){[char][int]"0x$_"}})

    Tuesday, March 03, 2015 2:54 PM
    Moderator
  • By the way, you were correct to continue this in the Virus and Malware forum on Microsoft Community.

    cf. OP's thread in Answers/Microsoft Community Consumer-specific forum: http://answers.microsoft.com/en-us/protect/forum/mse-protect_scanning/real-time-protection-causes-slowdowns-when/35bb4019-6d97-4f4b-be49-c96a0fbf215e

    ~Robear Dyer (PA Bear) MS MVP-Windows Client since 2002 Disclaimer: MS MVPs neither represent nor work for Microsoft

    Tuesday, March 03, 2015 7:04 PM