Penanya
w3wp.exe will hit 100% CPU after submit job, and job stuck in configuring status

Pertanyaan
-
Our application use IIS send HPC job. the current incident on client system w3wp.exe will hit 100% CPU.
the dump of w3wp.exe (during 100% CPU) has these thread:
most are HPC scheduler.Store related.
Any suggestion what could be wrong?
App
Windows Server 2016
HPC:
Windows Server 2016
HPC Version : 5.1.6086.0
HPCdatabase: is in external provider, we don’t have permission to check it. From CMDB I can see these:
Microsoft SQL Server 2012 SP3Not Flagged 992 0 Worker Thread Worker Thread Microsoft.Hpc.Scheduler.Store.dll!Microsoft.Hpc.Scheduler.Store.PropertyLookup.GetPropertyDescriptors 0
Not Flagged 9104 0 Worker Thread Worker Thread Microsoft.Hpc.Scheduler.Store.dll!Microsoft.Hpc.Scheduler.Store.PropertyLookup.GetPropertyDescriptors 0
Not Flagged 7672 0 Worker Thread Worker Thread Microsoft.Hpc.Scheduler.Store.dll!Microsoft.Hpc.Scheduler.Store.PropertyLookup.GetPropertyDescriptors 0
Not Flagged 6920 0 Worker Thread Worker Thread Microsoft.Hpc.Scheduler.Store.dll!Microsoft.Hpc.Scheduler.Store.PropertyLookup.GetPropertyDescriptors 0
Not Flagged 7544 0 Worker Thread clr.dll thread Microsoft.Hpc.Scheduler.Store.dll!Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.MonitorThread 0
Not Flagged 2264 0 Worker Thread clr.dll thread Microsoft.Hpc.Scheduler.Store.dll!Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.MonitorThread 0
Not Flagged 5732 0 Worker Thread clr.dll thread Microsoft.Hpc.Scheduler.Store.dll!Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.MonitorThread 0
Not Flagged 9652 0 Worker Thread clr.dll thread Microsoft.Hpc.Scheduler.Store.dll!Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.MonitorThread 0
Not Flagged 3856 0 Worker Thread clr.dll thread Microsoft.Hpc.Scheduler.Store.dll!Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.ReconnectThread 0
Not Flagged 9492 0 Worker Thread clr.dll thread Microsoft.Hpc.Scheduler.Store.dll!Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.ReconnectThread 0
Not Flagged 6524 0 Worker Thread clr.dll thread Microsoft.Hpc.Scheduler.Store.dll!Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.ReconnectThread 0
Not Flagged > 10152 0 Worker Thread clr.dll thread Microsoft.Hpc.Scheduler.Store.dll!Microsoft.Hpc.Scheduler.Store.SchedulerStoreSvc.ReconnectThread 0
Not Flagged 9364 0 Worker Thread clr.dll thread mswsock.dll!00007fff09348033 0
Not Flagged 1520 0 Worker Thread clr.dll thread mswsock.dll!00007fff09348033 0
Not Flagged 4056 0 Worker Thread clr.dll thread mswsock.dll!00007fff09348033 0
Not Flagged 4328 0 Worker Thread clr.dll thread mswsock.dll!00007fff09348033 0
Not Flagged 2056 0 Worker Thread nativerd.dll thread nativerd.dll!00007ffefa0259a8 0
Not Flagged 5488 0 Worker Thread clr.dll thread ntdll.dll!00007fff0d6c5b94 0
Not Flagged 5808 0 Worker Thread clr.dll thread ntdll.dll!00007fff0d6c5b94 0
Not Flagged 8372 0 Worker Thread clr.dll thread ntdll.dll!00007fff0d6c5b94 0
Not Flagged 8796 0 Worker Thread clr.dll thread ntdll.dll!00007fff0d6c5b94 0
Not Flagged 700 0 Worker Thread clr.dll thread ntdll.dll!00007fff0d6c5b94 0
Not Flagged 5284 0 Worker Thread clr.dll thread ntdll.dll!00007fff0d6c5b94 0
Not Flagged 4208 0 Worker Thread clr.dll thread ntdll.dll!00007fff0d6c5b94 0
Not Flagged 688 0 Worker Thread clr.dll thread ntdll.dll!00007fff0d6c5b94 0
Not Flagged 4572 0 Worker Thread clr.dll thread ntdll.dll!00007fff0d6c5b94 0
Not Flagged 5748 0 Worker Thread clr.dll thread ntdll.dll!00007fff0d6c5b94 0
Not Flagged 9872 0 Worker Thread clr.dll thread ntdll.dll!00007fff0d6c5b94 0
Not Flagged 8616 0 Worker Thread clr.dll thread ntdll.dll!00007fff0d6c5b94 0
Not Flagged 3964 0 Worker Thread clr.dll thread ntdll.dll!00007fff0d6c5b94 0
Not Flagged 7584 0 Worker Thread clr.dll thread ntdll.dll!00007fff0d6c5b94 0
Not Flagged 8368 0 Worker Thread clr.dll thread ntdll.dll!00007fff0d6c5b94 0
Not Flagged 5236 0 Worker Thread clr.dll thread ntdll.dll!00007fff0d6c5c34 0
Not Flagged 9896 0 Worker Thread clr.dll thread ntdll.dll!00007fff0d6c6194 0
Not Flagged 4220 0 Worker Thread clr.dll thread ntdll.dll!00007fff0d6c6664 0
Not Flagged 8488 0 Worker Thread clr.dll thread ntdll.dll!00007fff0d6c6664 0
Not Flagged 8032 0 Worker Thread clr.dll thread ntdll.dll!00007fff0d6c6664 0
Not Flagged 1656 0 Worker Thread clr.dll thread ntdll.dll!00007fff0d6c71c4 0
Not Flagged 2608 0 Worker Thread ntdll.dll thread ntdll.dll!00007fff0d6c92e4 0
Not Flagged 8024 0 Worker Thread ntdll.dll thread ntdll.dll!00007fff0d6c92e4 0
Not Flagged 9604 0 Worker Thread ntdll.dll thread ntdll.dll!00007fff0d6c92e4 0
Not Flagged 7660 0 Worker Thread ntdll.dll thread ntdll.dll!00007fff0d6c92e4 0
Not Flagged 1360 0 Worker Thread rasman.dll thread rasman.dll!00007ffeecf1bbb6 0
Not Flagged 4492 0 Worker Thread sqlncli11.dll thread sqlncli11.dll!00007ffee0e40f04 0
Not Flagged 2912 0 Worker Thread System.Data.dll thread System.Data.dll!00007ffee3008974 0
Not Flagged 32 0 Worker Thread clr.dll thread System.DirectoryServices.Protocols.dll!System.DirectoryServices.Protocols.PartialResultsRetriever.ThreadRoutine 0
Not Flagged 8340 0 Worker Thread w3tp.dll thread w3tp.dll!00007ffeff77181a 0
Not Flagged 4252 0 Worker Thread w3tp.dll thread w3tp.dll!00007ffeff77181a 0
Not Flagged 8052 0 Worker Thread w3tp.dll thread w3tp.dll!00007ffeff77181a 0
Not Flagged 804 0 Worker Thread w3tp.dll thread w3tp.dll!00007ffeff77181a 0
Not Flagged 7812 0 Main Thread Main Thread w3wphost.dll!00007fff01d018ac 0
Rabu, 18 Desember 2019 21.46
Semua Balasan
-
Hi Yingqun,
Is this 100% CPU consumption constant and is there any connection issue for the SQL DB? The IIS and w3wp.exe are running on the head node, correct? Can you just submit a simple job on the head node, e.g. 'job submit hostname' to validate the scheduler service and SQL DB?
Also note we have a known issue for scheduler connection leak for HPC Web portal in HPC Pack 2016 Update 1 (5.1). If possible, could you upgrade the cluster to Update 3 (5.3) and retry?
Regards,
Yutong Sun
Rabu, 25 Desember 2019 15.02