locked
resolving task sequence dependencies slow since SCCM CB 2002 Update RRS feed

  • Vraag

  • Hi,

    We've just upgraded from SCCM CB 1910 to 2002 over the weekend and since then in our OS builds on the 'resolving task sequence dependencies' step its extremely slow.

    What felt like a minute or so processing this step now takes approx 2hrs. The builds still complete but very slow.

    I've just logged a premier support ticket to help troubleshoot this but I am interested if anyone else has experienced anything similar or suggestions on what could possibly slow this step down that I could look at?

    Thanks.


    dinsdag 19 mei 2020 12:02

Alle reacties

  • Hi,

    Thanks for posting in TechNet.

    It's firstly recommended to review the smsts.log to see if there is any clue.

    Please help check the anti-virus software. And if you have do the Endpoint Protection update during OSD, please remove it to have a try. Similar threads for your reference:
    Resolving task sequence dependencies taking long time
    Slow OSD on one DP

    Thanks for your time.

    Best regards,
    Simon

    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    woensdag 20 mei 2020 03:13
  • Hi Simon,

    The SMSTS logs don't show any errors occurring, apart from taking several seconds between each get policy request...feels like SQL or the MP slow to respond with the data.

    We don't use Endpoint Protection, but might explore disabling AV on the SQL or MP servers.

    Its interesting in that we have 3 environments, DEV, UAT and PROD. In DEV there is no issue but UAT and PROD do.

    woensdag 20 mei 2020 12:33
  • Hi,

    Thanks for your reply. 

    It's really strange issue. Are there any difference about network configuration between them? If possible, please recreate an identical task sequence deployed to AT and PROD environment to have a try. 

    Thanks for your time. 
     
    Best regards,
    Simon

    Please remember to mark the replies as answers if they help.
    If you have feedback for TechNet Subscriber Support, contact tnmff@microsoft.com.

    donderdag 21 mei 2020 10:44
  • I'd be interested in hearing if you ever got a resolution on this?  We just did our 1910 to 2002 upgrade over the past weekend an immediately go this same complaint from our support staff this morning. First complaints were saying it was taking 1-2 hours to check the TS dependencies, but my testing has so far been better at 30-35 minutes. However prior to upgrading we were seeing more like 5-10 minutes for the same general task sequence. Yes, we've made a few changes in the TS, but just changing a few installed packages. Once the check is done the rest runs normally.

    Similarly to what you said, there are no problems reported in SSMTS.log or IIS logs on the MP during the process. It's just slow. Averaging around 1-2 seconds per packageID based on sampling data from the SMSTS.log. We also upgraded our DEV environment several weeks ago and did not notice this issue there either.


    • Bewerkt door Not_IT maandag 1 juni 2020 22:32
    maandag 1 juni 2020 22:30
  • Hi,

    Kinda feel better someone else has this issue and not just me lol. MS have said they have no one else reporting this.

    Issue still ongoing but seems getting closer to the root cause. Numerous logs and profiling across SQL, MP, Procdumps and IIS occuring with MS Support. 

    But it is slow going with different timezones in the mix.

    Our SQL server seems busier since the upgrade and after we increased memory and CPU its handling it better but still slow very slow. MS aren't suggesting this is the root cause but its eased some of the constraints.

    As soon as I get something more concrete I will post what that is.

    maandag 1 juni 2020 23:22
  • Hi,

    first of all, sorry for my bad englisch.

    Your are not alone. We just did our 1910 to 2002 upgrade on Friday 29.05.2020 and have the same issues.   Start the device-installation via Boot-Image, resolving TS Policy dependencies round about 3 hrs. (->a lot of Steps in the TS; 30sec for each PackageID request).

    (By the way: our PXe-Server is out of work after SCCM Upgrade. Reinstall with many differnce step by step guides, without success. Logfile seems to be stranged. Between "DHCP Discover received" and "Device is in the database" 30sec. Between "Device is in the database" and "found optional advertisement" 1min.).

    Similarly to what you said, there are no problems reported in SSMTS.log or IIS logs on the MP during the process.

    Run a TS via SoftwareCenter, there is no problem!

    Now we have opened a premier support ticket, too.

    dinsdag 2 juni 2020 10:15
  • Hi,

    PXE is broken for us as well and also logged a support ticket. The same engineer assigned to our TS slowness is also handling the PXE issue and i believe its a common root cause...fix the slowness and PXE will also be resolved.

    dinsdag 2 juni 2020 11:36
  • Hi, we are also affected by this very same topic. Currently under investigation via MS engineer from premiere support is a change related to following stored procedure: sp_GetPublicKeyForSMSID

    This one can be observed to be called in unusual high amounts, also causing our SQL server to be very busy (CPU Util goes up to 100% for longer times since upgrading). Top talkers causing the sp to be called are clients running our imaging task sequence.

    Run following SQL statement to get the execution count of mentioned sp, maybe confirming it to be the top talker:

    SELECT TOP 30 d.object_id, d.database_id, OBJECT_NAME(object_id, database_id) 'proc name', d.cached_time, d.last_execution_time, d.total_elapsed_time, d.total_elapsed_time/d.execution_count AS [avg_elapsed_time], d.last_elapsed_time, d.execution_count 
    FROM sys.dm_exec_procedure_stats AS d ORDER BY [total_worker_time] DESC;


    Beside this one, you can also run the SQL report "Performance - Top Queries by Total CPU Time", eventually helping you to nail it down even further.

    If I have some more progress out of our case, I'll kindly post another update. Looking forward to yours, too!

    maandag 8 juni 2020 06:37
  • Oh, and would you like to share, what your hierarchy configuration is like in terms of the "communications" tab? Here's ours, and the "Use PKI client certificate..." check box is suspected to be the culprit, but we did not change it yet:

    maandag 8 juni 2020 07:00
  • Hi,

    So far from MS they were looking at delays seemingly from IIS and RPC calls from ccmisapi.dll. They would get me to test various SP calls as well which would always return in less the 1sec. They said they are going to arrange a remote session with a SQL specialist so it may end up similar to your findings.

    I had noticed 'sp_GetPublicKeyForSMSID' as well but i didn't know if that was being called or more expensive than prior to the upgrade.

    I'm not at work at the moment to get a screenshot but we are using the Enhanced HTTPS feature but pretty sure don't have the CRL check box enabled. 

    When i'm back at work tomorrow i'll run those queries for comparison and get a screenshot.

    maandag 8 juni 2020 08:11
  • Same issue after update last week.  Hours to resolve TS dependencies.  PXE problems.  Opened premiere case.
    • Bewerkt door arkaine23 maandag 8 juni 2020 13:35
    maandag 8 juni 2020 13:35
  • @Uhle,

    Thanks for the additional details. Our communication settings are set exactly the same as yours.  We aren't seeing quite as dramatic of a rise in CPU % on our SQL server (separate dedicated server), but it does go moderately higher when task sequences are running. 

    I'm running a task sequence now that is resolving dependencies now and do see sp_GetPublicKeyForSMSID climbing up the list as the procedure using the most worker time. Likewise, I don't have detailed knowledge of the kinds of stats this procedure had  <= 1910 compared to 2002 so it's just an observation.  



    maandag 8 juni 2020 14:33
  • Hi,

    Just confirmed my site's communication settings and its the same as yours apart from us using Enhanced HTTPS.

    Also ran your suggested query and these are the top 2 in order:

    MP_GetContentDPInfo
    sp_GetPublicKeyForSMSID


    • Bewerkt door mgrant3 maandag 8 juni 2020 23:42
    maandag 8 juni 2020 23:41
  • Hi,

    Expanding on that 'sp_GetPublicKeyForSMSID' finding, MS suggested the following.

    In the previous SCCM release there is a part of the SP like this:

    SELECT 'Y'
                           FROM   MachineIdGroupXRef m
                                  INNER JOIN ResPolicyMap rpm ON m.MachineID = rpm.MachineID
                                  INNER JOIN SoftwarePolicy sp ON sp.PADBID = rpm.PADBID
                                  INNER JOIN PkgPrograms pp ON pp.PkgID = sp.PkgID AND sp.ProgramName = pp.Name
                                  INNER JOIN TS_TaskSequence ts ON pp.ProgramID = ts.TS_ID
                           WHERE  m.GUID = @SMSID
                         


    On SCCM 2002, it is :

    SELECT 'Y' 
                           FROM   MachineIdGroupXRef m 
                                  INNER JOIN ResPolicyMap rpm ON m.MachineID = rpm.MachineID 
                                  INNER JOIN SoftwarePolicy sp ON sp.PADBID = rpm.PADBID 
                                  INNER JOIN PkgPrograms pp ON pp.PkgID = sp.PkgID AND sp.ProgramName = pp.Name 
                                  INNER JOIN TS_TaskSequence ts ON pp.ProgramID = ts.TS_ID 
                           WHERE  m.GUID = @SMSID OR m.GUID in (select TOP 1 SMS_Unique_Identifier0 from ProvisioningSystem_DISC) 

    MS suggested to remove the 'OR m.GUID in (select TOP 1 SMS_Unique_Identifier0 from ProvisioningSystem_DISC)' part in our UAT environment which i've now done and its brought performance back to normal.

    Now confirming with MS what breaks or impacted by removing this entry and the next recommendations.

    dinsdag 9 juni 2020 02:27
  • Hi,

    Final iteration of this workaround that MS recommended us to apply is as follows:

    Run this command:

    CREATE NONCLUSTERED INDEX [MSFT_SMSIDSPIssue] ON  dbo.SoftwarePolicy ([PKGID], [ProgramName]) INCLUDE ([PADBID])

    Modify the SP 'sp_GetPublicKeyForSMSID' where it says 'UNION' to 'UNION ALL' as below:

    IF EXISTS( SELECT 'Y' 
                           FROM   MachineIdGroupXRef m 
                                  INNER JOIN ResPolicyMap rpm ON m.MachineID = rpm.MachineID 
                                  INNER JOIN SoftwarePolicy sp ON sp.PADBID = rpm.PADBID 
                                  INNER JOIN PkgPrograms pp ON pp.PkgID = sp.PkgID AND sp.ProgramName = pp.Name 
                                  INNER JOIN TS_TaskSequence ts ON pp.ProgramID = ts.TS_ID 
                           WHERE  m.GUID = @SMSID OR m.GUID in (select TOP 1 SMS_Unique_Identifier0 from ProvisioningSystem_DISC) 
                    UNION ALL
                    SELECT 'Y' 
                           FROM   PkgPrograms pp  
                                  INNER JOIN TS_TaskSequence ts ON pp.ProgramID = ts.TS_ID 
                           WHERE  pp.ProgramFlags & 1 <> 0 
                ) 

    We've seen a huge improvement with this, not sure if its at the same performance level as before upgrade but much better than current.

    dinsdag 9 juni 2020 07:25
  • Hi,

    after getting a Critical Situation Manager this morning, his Technical Engineer gives us exactly the same workaround as mgrant3.

    Now everthing works, but we should monitor that...

    dinsdag 9 juni 2020 08:15
  • Hi Mgrant3, Tymi01

    Please work with your CSS engineers to collect more details on which exact change causes the effect: index or UNION ALL or something else.

    dinsdag 9 juni 2020 12:25
  • Just wrapped up with our CSS Engineer for this issue and can say all we changed was this line in sp_GetPublicKeyForSMSID that @mgrant3 noted above and it has solved the slow OSD dependency issue:

    OLD:  WHERE  m.GUID = @SMSID OR m.GUID in (select TOP 1 SMS_Unique_Identifier0 from ProvisioningSystem_DISC) 

    NEW:  WHERE  m.GUID = @SMSID

    We don't currently use PXE (for a different reason) so I can't say if it helps there though.

    dinsdag 9 juni 2020 18:50
  • Same issue after update last weekend.  Hours to resolve TS dependencies. Sometimes even failing.  PXE problems. Opened case.
    • Bewerkt door Crasser woensdag 10 juni 2020 06:38
    woensdag 10 juni 2020 06:38
  • Whoever finds this thread, please install the following hotfix for ConfigMgr 2002: https://support.microsoft.com/en-us/help/4567007/pxe-boot-fails-after-updating-to-configuration-manager-current-branch

    Please note that the issue is specific for 2002 version, so if you experience similar symptoms, please log the case with Microsoft.

    • Als antwoord voorgesteld door Sergey Korotkov woensdag 17 juni 2020 08:34
    woensdag 17 juni 2020 06:03