locked
Workflows with "Wait" action stuck in "in progress" RRS feed

  • Question

  • Hey all,

    I have a very odd problem at a customer site. First some background information:

    We've created a CRM system which supports our customer from lead to order and from order to (install to) completion processes. This (ofcourse) mainly uses the sales module. To support the processes above they use alot of workflows. Alot of these workflows have Wait actions in them: wait until task is completed, wait until attribute contains a value, or just wait for n time. This is all hosted on a shared SaaS platform with an IFD deployment.

    Recently they had some issues with mail not being sent out properly from CRM. This seemed to have lasted for nearly two months until someone discovered it. This caused a whopping 144712 e-mails to get stuck in "Pending send," our SaaS provider also manages the accounts and the e-mail router. When we created the ticket it took a week or two until they fixed the issue. It seemed that the e-mail router was not configured to let the app pool user send the e-mail requests resulting in a whole lot of 401 errors. They 'just' fixed this without informing us first causing the server to become rather busy with sending e-mails, marking the start of our worries.

    The problem right now is that our a-sync service is being filled up with workflows. Most of these workflows are on the "waiting" action, but they aren't being set in to the waiting state, instead they all just idle in the "In progress" state. This seems to fill up the queue of workflows completely using up over 2 GB (of the available 4), and using up a minimum of 50% till 80-90% of the CPU time. After a while the async service just dies, starts back up again and within about 5 minutes it fills up the memory completely again. Trace logs don't show any errors, most of it would be pretty much expected activity. Workflows starting up and then being put away again after being 'idle for too long.'

    One of our major problems in diagnosing is that we can't get on to the SQL database since it's a shared platform, but still all diagnostics including the ones that involve SQL queries would be much appreciated. We'll still try and get access to the database throughout the day since the system is unusable right now for our customer.

    - Alex

    [edit]
    We've just updated to Rollup 10, this did not fix the issue :)
    Also: to battle this issue we've removed all of the e-mails that used to be stuck in "Pending send" this also did not result in any type of improvement.
    [/edit]

    • Edited by Alex Ries Tuesday, May 18, 2010 7:49 AM Additional Information
    Tuesday, May 18, 2010 6:22 AM

Answers

  • Do you tried to clear asyncoperation table. It will delete all the workflows but your system will be stable.

     

    http://support.microsoft.com/kb/968520


    My Dynamics CRM Blog: http://bovoweb.blogspot.com
    Tuesday, May 18, 2010 10:24 AM
  • Hi Alex,

    Definitely look into kgorczewski's recommendation as that will help performance.  Once you clear the asyn table, it should automatically clear moving forward since you are on UR 10.

    there are a couple of things you should take into consideration when using a Wait condition.  You generally need to add some additional processes to your workflows to ensure the system jobs spawned from the workflow don't sit in a permanent Wait Condition.  I'll provide one example and you can have a look at my post for additional information.

    Example: let's say you have a workflow that waits for an update to the Sales Stage code field.  However, the record is closed before the field value changes.  If you don't have a step in the workflow to loop back through every day and check to ensure the record is still active, then that system job will stay in the Wait status.  A better way to create this type of workflow is to use the workflow option, Field Value Change rather than Wait condition.

    There may be some things you can do to streamline your workflows and make them more efficient.

     


    Regards, Donna

            Windows Live Blog

    Tuesday, May 18, 2010 1:16 PM

All replies

  • Do you tried to clear asyncoperation table. It will delete all the workflows but your system will be stable.

     

    http://support.microsoft.com/kb/968520


    My Dynamics CRM Blog: http://bovoweb.blogspot.com
    Tuesday, May 18, 2010 10:24 AM
  • Hi Alex,

    Definitely look into kgorczewski's recommendation as that will help performance.  Once you clear the asyn table, it should automatically clear moving forward since you are on UR 10.

    there are a couple of things you should take into consideration when using a Wait condition.  You generally need to add some additional processes to your workflows to ensure the system jobs spawned from the workflow don't sit in a permanent Wait Condition.  I'll provide one example and you can have a look at my post for additional information.

    Example: let's say you have a workflow that waits for an update to the Sales Stage code field.  However, the record is closed before the field value changes.  If you don't have a step in the workflow to loop back through every day and check to ensure the record is still active, then that system job will stay in the Wait status.  A better way to create this type of workflow is to use the workflow option, Field Value Change rather than Wait condition.

    There may be some things you can do to streamline your workflows and make them more efficient.

     


    Regards, Donna

            Windows Live Blog

    Tuesday, May 18, 2010 1:16 PM