locked
w3wp.exe will hit 100% CPU after submit job, and job stuck in configuring status RRS feed

  • Question

  • Our application use IIS send HPC job. the current incident on client system w3wp.exe will hit 100% CPU.

    the dump of w3wp.exe (during 100% CPU) has these thread: 

    most are HPC scheduler.Store related.

    Any suggestion what could be wrong? 

    App
    Windows Server 2016

    HPC:
    Windows Server 2016
    HPC Version              : 5.1.6086.0

    HPCdatabase: is in external provider, we don’t have permission to check it. From CMDB I can see these:
    Microsoft SQL Server 2012 SP3

    Not Flagged             992     0          Worker Thread       Worker Thread Microsoft.Hpc.Scheduler.Store.dll!Microsoft.Hpc.Scheduler.Store.PropertyLookup.GetPropertyDescriptors            0

    Not Flagged             9104  0          Worker Thread       Worker Thread Microsoft.Hpc.Scheduler.Store.dll!Microsoft.Hpc.Scheduler.Store.PropertyLookup.GetPropertyDescriptors            0

    Not Flagged             7672  0          Worker Thread       Worker Thread Microsoft.Hpc.Scheduler.Store.dll!Microsoft.Hpc.Scheduler.Store.PropertyLookup.GetPropertyDescriptors            0

    Not Flagged             6920  0          Worker Thread       Worker Thread Microsoft.Hpc.Scheduler.Store.dll!Microsoft.Hpc.Scheduler.Store.PropertyLookup.GetPropertyDescriptors            0

    Not Flagged             7544  0          Worker Thread       clr.dll thread Microsoft.Hpc.Scheduler.Store.dll!Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.MonitorThread            0

    Not Flagged             2264  0          Worker Thread       clr.dll thread Microsoft.Hpc.Scheduler.Store.dll!Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.MonitorThread            0

    Not Flagged             5732  0          Worker Thread       clr.dll thread Microsoft.Hpc.Scheduler.Store.dll!Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.MonitorThread            0

    Not Flagged             9652  0          Worker Thread       clr.dll thread Microsoft.Hpc.Scheduler.Store.dll!Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.MonitorThread            0

    Not Flagged             3856  0          Worker Thread       clr.dll thread Microsoft.Hpc.Scheduler.Store.dll!Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.ReconnectThread            0

    Not Flagged             9492  0          Worker Thread       clr.dll thread Microsoft.Hpc.Scheduler.Store.dll!Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.ReconnectThread            0

    Not Flagged             6524  0          Worker Thread       clr.dll thread Microsoft.Hpc.Scheduler.Store.dll!Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.ReconnectThread            0

    Not Flagged >          10152 0          Worker Thread       clr.dll thread Microsoft.Hpc.Scheduler.Store.dll!Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.ReconnectThread            0

    Not Flagged             9364  0          Worker Thread       clr.dll thread            mswsock.dll!00007fff09348033 0

    Not Flagged             1520  0          Worker Thread       clr.dll thread            mswsock.dll!00007fff09348033 0

    Not Flagged             4056  0          Worker Thread       clr.dll thread            mswsock.dll!00007fff09348033 0

    Not Flagged             4328  0          Worker Thread       clr.dll thread            mswsock.dll!00007fff09348033 0

    Not Flagged             2056  0          Worker Thread       nativerd.dll thread            nativerd.dll!00007ffefa0259a8  0

    Not Flagged             5488  0          Worker Thread       clr.dll thread            ntdll.dll!00007fff0d6c5b94         0

    Not Flagged             5808  0          Worker Thread       clr.dll thread            ntdll.dll!00007fff0d6c5b94         0

    Not Flagged             8372  0          Worker Thread       clr.dll thread            ntdll.dll!00007fff0d6c5b94         0

    Not Flagged             8796  0          Worker Thread       clr.dll thread            ntdll.dll!00007fff0d6c5b94         0

    Not Flagged             700     0          Worker Thread       clr.dll thread            ntdll.dll!00007fff0d6c5b94         0

    Not Flagged             5284  0          Worker Thread       clr.dll thread            ntdll.dll!00007fff0d6c5b94         0

    Not Flagged             4208  0          Worker Thread       clr.dll thread            ntdll.dll!00007fff0d6c5b94         0

    Not Flagged             688     0          Worker Thread       clr.dll thread            ntdll.dll!00007fff0d6c5b94         0

    Not Flagged             4572  0          Worker Thread       clr.dll thread            ntdll.dll!00007fff0d6c5b94         0

    Not Flagged             5748  0          Worker Thread       clr.dll thread            ntdll.dll!00007fff0d6c5b94         0

    Not Flagged             9872  0          Worker Thread       clr.dll thread            ntdll.dll!00007fff0d6c5b94         0

    Not Flagged             8616  0          Worker Thread       clr.dll thread            ntdll.dll!00007fff0d6c5b94         0

    Not Flagged             3964  0          Worker Thread       clr.dll thread            ntdll.dll!00007fff0d6c5b94         0

    Not Flagged             7584  0          Worker Thread       clr.dll thread            ntdll.dll!00007fff0d6c5b94         0

    Not Flagged             8368  0          Worker Thread       clr.dll thread            ntdll.dll!00007fff0d6c5b94         0

    Not Flagged             5236  0          Worker Thread       clr.dll thread            ntdll.dll!00007fff0d6c5c34         0

    Not Flagged             9896  0          Worker Thread       clr.dll thread            ntdll.dll!00007fff0d6c6194         0

    Not Flagged             4220  0          Worker Thread       clr.dll thread            ntdll.dll!00007fff0d6c6664         0

    Not Flagged             8488  0          Worker Thread       clr.dll thread            ntdll.dll!00007fff0d6c6664         0

    Not Flagged             8032  0          Worker Thread       clr.dll thread            ntdll.dll!00007fff0d6c6664         0

    Not Flagged             1656  0          Worker Thread       clr.dll thread            ntdll.dll!00007fff0d6c71c4         0

    Not Flagged             2608  0          Worker Thread       ntdll.dll thread            ntdll.dll!00007fff0d6c92e4         0

    Not Flagged             8024  0          Worker Thread       ntdll.dll thread            ntdll.dll!00007fff0d6c92e4         0

    Not Flagged             9604  0          Worker Thread       ntdll.dll thread            ntdll.dll!00007fff0d6c92e4         0

    Not Flagged             7660  0          Worker Thread       ntdll.dll thread            ntdll.dll!00007fff0d6c92e4         0

    Not Flagged             1360  0          Worker Thread       rasman.dll thread            rasman.dll!00007ffeecf1bbb6   0

    Not Flagged             4492  0          Worker Thread       sqlncli11.dll thread            sqlncli11.dll!00007ffee0e40f04 0

    Not Flagged             2912  0          Worker Thread       System.Data.dll thread            System.Data.dll!00007ffee3008974    0

    Not Flagged             32       0          Worker Thread       clr.dll thread System.DirectoryServices.Protocols.dll!System.DirectoryServices.Protocols.PartialResultsRetriever.ThreadRoutine            0

    Not Flagged             8340  0          Worker Thread       w3tp.dll thread            w3tp.dll!00007ffeff77181a        0

    Not Flagged             4252  0          Worker Thread       w3tp.dll thread            w3tp.dll!00007ffeff77181a        0

    Not Flagged             8052  0          Worker Thread       w3tp.dll thread            w3tp.dll!00007ffeff77181a        0

    Not Flagged             804     0          Worker Thread       w3tp.dll thread            w3tp.dll!00007ffeff77181a        0

    Not Flagged             7812  0          Main Thread            Main Thread            w3wphost.dll!00007fff01d018ac          0

    Wednesday, December 18, 2019 9:46 PM

All replies

  • Hi Yingqun,

    Is this 100% CPU consumption constant and is there any connection issue for the SQL DB? The IIS and w3wp.exe are running on the head node, correct? Can you just submit a simple job on the head node, e.g. 'job submit hostname' to validate the scheduler service and SQL DB?

    Also note we have a known issue for scheduler connection leak for HPC Web portal in HPC Pack 2016 Update 1 (5.1). If possible, could you upgrade the cluster to Update 3 (5.3) and retry?

    Regards,

    Yutong Sun

    Wednesday, December 25, 2019 3:02 PM