locked
What is the memory limit for a plug-in in sandbox isolation RRS feed

  • Question

  • I am troubleshooting a scenario where a plug-in is failing due to what I suspect is a memory limit. The plugin is triggered on the creation of an annotation, and the plugin fails when the annotation includes an attachment about 10 MB and larger. Testing to narrow down the breaking point has found a 10,159 KB attachment works fine but a 10,183 KB attachment triggers the failure. The MSDN page Plug-in isolation, trusts, and statistics states that sandbox worker processes are terminated when exceeding memory limits:

    Microsoft Dynamics CRM collects run-time statistics and monitors plug-ins and custom workflow activities that execute in the sandbox. If the sandbox worker process that hosts this custom code exceeds threshold CPU, memory, or handle limits or is otherwise unresponsive, that process will be killed by the platform. At that point any currently executing plug-in or custom workflow activity in that worker process will fail with exceptions. However, the next time that the plug-in or custom workflow activity is executed it will run normally. There is one worker process per organization so failures in one organization will not affect another organization.

    Changing the plugin's isolation mode to None does remove the error, but I'd like to know more about this limit. What is the exact memory limit, and is there a setting to change it?





    • Edited by Alan.M Friday, August 14, 2015 6:33 PM
    Friday, August 14, 2015 6:19 PM

All replies

  • Alan - what is the exact error text you are getting ? Have you tried enabling tracing - what do you see in there can you post what is happening near the error?

    .NET MVP / Microsoft Regional Director / http://blog.davidyack.com / http://crm.davidyack.com

    Friday, August 28, 2015 3:25 PM
  • I have reproduced the error on a separate system than the one the error was originally occurring on and gathered tracing information.

    This is the message shown in the system job:

    The plug-in execution failed because no Sandbox Worker processes are currently available. Please try again.

    System.ServiceModel.CommunicationException: Microsoft Dynamics CRM has experienced an error. Reference number for administrators or support: #785CC4E2

    This is the exception in the system job:

    Plugin Trace:


    [Adxstudio.Xrm.Plugins: Adxstudio.Xrm.Plugins.WebNotificationPlugin]
    [54bb256e-e710-e311-9429-00155d03690b: Adxstudio.Xrm.Plugins.WebNotificationPlugin: Create of  any Entity]

    Error Message:

    Unhandled Exception: System.ServiceModel.FaultException`1[[Microsoft.Xrm.Sdk.OrganizationServiceFault, Microsoft.Xrm.Sdk, Version=7.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]]: The plug-in execution failed because no Sandbox Worker processes are currently available. Please try again.
    System.ServiceModel.CommunicationException: Microsoft Dynamics CRM has experienced an error. Reference number for administrators or support: #785CC4E2Detail:
    <OrganizationServiceFault xmlns:i="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schemas.microsoft.com/xrm/2011/Contracts">
      <ErrorCode>-2147204723</ErrorCode>
      <ErrorDetails xmlns:d2p1="http://schemas.datacontract.org/2004/07/System.Collections.Generic" />
      <Message>The plug-in execution failed because no Sandbox Worker processes are currently available. Please try again.
    System.ServiceModel.CommunicationException: Microsoft Dynamics CRM has experienced an error. Reference number for administrators or support: #785CC4E2</Message>
      <Timestamp>2015-08-28T17:20:34.3579864Z</Timestamp>
      <InnerFault>
        <ErrorCode>-2147220970</ErrorCode>
        <ErrorDetails xmlns:d3p1="http://schemas.datacontract.org/2004/07/System.Collections.Generic" />
        <Message>System.IO.IOException: Microsoft Dynamics CRM has experienced an error. Reference number for administrators or support: #E4BFBE34</Message>
        <Timestamp>2015-08-28T17:20:34.3579864Z</Timestamp>
        <InnerFault>
          <ErrorCode>-2147204723</ErrorCode>
          <ErrorDetails xmlns:d4p1="http://schemas.datacontract.org/2004/07/System.Collections.Generic" />
          <Message>The plug-in execution failed because no Sandbox Worker processes are currently available. Please try again.
    System.ServiceModel.CommunicationException: Microsoft Dynamics CRM has experienced an error. Reference number for administrators or support: #928EF490</Message>
          <Timestamp>2015-08-28T17:20:34.3736018Z</Timestamp>
          <InnerFault>
            <ErrorCode>-2147220970</ErrorCode>
            <ErrorDetails xmlns:d5p1="http://schemas.datacontract.org/2004/07/System.Collections.Generic" />
            <Message>System.IO.PipeException: Microsoft Dynamics CRM has experienced an error. Reference number for administrators or support: #C017D5E8</Message>
            <Timestamp>2015-08-28T17:20:34.3736018Z</Timestamp>
            <InnerFault i:nil="true" />
            <TraceText i:nil="true" />
          </InnerFault>
          <TraceText i:nil="true" />
        </InnerFault>
        <TraceText i:nil="true" />
      </InnerFault>
      <TraceText>
    [Adxstudio.Xrm.Plugins: Adxstudio.Xrm.Plugins.WebNotificationPlugin]
    [54bb256e-e710-e311-9429-00155d03690b: Adxstudio.Xrm.Plugins.WebNotificationPlugin: Create of  any Entity]

    </TraceText>
    </OrganizationServiceFault>

    These platform tracing files (OneDrive links) have similar exception information in them:

    This platform tracing file from the sandbox worker process has an additional exception about a message size quota being exceeded that could be the originating source of the error:

    System.ServiceModel.QuotaExceededException: The maximum message size quota for incoming messages (27787264) has been exceeded. To increase the quota, use the MaxReceivedMessageSize property on the appropriate binding element.

    [2015-08-28 11:20:34.357] Process:Microsoft.Crm.Sandbox.WorkerProcess |Organization:00000000-0000-0000-0000-000000000000 |Thread:   11 |Category: Sandbox |User: 00000000-0000-0000-0000-000000000000 |Level: Warning |ReqId: 00000000-0000-0000-0000-000000000000 | SandboxErrorHandler.HandleError  ilOffset = 0xE
     at SandboxErrorHandler.HandleError(Exception error)  ilOffset = 0xE
     at ErrorBehavior.HandleErrorCommon(Exception error, ErrorHandlerFaultInfo& faultInfo)  ilOffset = 0x38
     at ChannelDispatcher.HandleError(Exception error, ErrorHandlerFaultInfo& faultInfo)  ilOffset = 0x35
     at ChannelDispatcher.HandleError(Exception error)  ilOffset = 0xC
     at ErrorHandlingReceiver.EndTryReceive(IAsyncResult result, RequestContext& requestContext)  ilOffset = 0x21
     at ChannelHandler.EndTryReceive(IAsyncResult result, RequestContext& requestContext)  ilOffset = 0x21
     at ChannelHandler.AsyncMessagePump(IAsyncResult result)  ilOffset = 0x2E
     at AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)  ilOffset = 0x0
     at AsyncResult.Complete(Boolean completedSynchronously)  ilOffset = 0xC2
     at TryReceiveAsyncResult.OnReceive(IAsyncResult result)  ilOffset = 0x4B
     at AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)  ilOffset = 0x0
     at AsyncResult.Complete(Boolean completedSynchronously)  ilOffset = 0xC2
     at ReceiveAsyncResult.OnReceiveComplete(Object state)  ilOffset = 0x2E
     at SessionConnectionReader.OnAsyncReadComplete(Object state)  ilOffset = 0xBF
     at AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)  ilOffset = 0x0
     at LazyAsyncResult.Complete(IntPtr userToken)  ilOffset = 0x3E
     at NegotiateStream.ProcessFrameBody(Int32 readBytes, Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)  ilOffset = 0x70
     at NegotiateStream.ReadCallback(AsyncProtocolRequest asyncRequest)  ilOffset = 0x68
     at FixedSizeReader.CheckCompletionBeforeNextRead(Int32 bytes)  ilOffset = 0x5D
     at FixedSizeReader.ReadCallback(IAsyncResult transportResult)  ilOffset = 0x29
     at AsyncResult.Complete(Boolean completedSynchronously)  ilOffset = 0xC2
     at IOAsyncResult.OnAsyncIOComplete(Object state)  ilOffset = 0x29
     at PipeConnection.OnAsyncReadComplete(Boolean haveResult, Int32 error, Int32 numBytes)  ilOffset = 0x143
     at OverlappedContext.CompleteCallback(UInt32 error, UInt32 numBytes, NativeOverlapped* nativeOverlapped)  ilOffset = 0x4C
     at IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)  ilOffset = 0x5
     at _IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)  ilOffset = 0x3C
    >System.ServiceModel.CommunicationException: The maximum message size quota for incoming messages (27787264) has been exceeded. To increase the quota, use the MaxReceivedMessageSize property on the appropriate binding element. ---> System.ServiceModel.QuotaExceededException: The maximum message size quota for incoming messages (27787264) has been exceeded. To increase the quota, use the MaxReceivedMessageSize property on the appropriate binding element.
       --- End of inner exception stack trace ---
       at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
       at System.ServiceModel.Channels.TransportDuplexSessionChannel.TryReceiveAsyncResult.End(IAsyncResult result, Message& message)
       at System.ServiceModel.Dispatcher.DuplexChannelBinder.EndTryReceive(IAsyncResult result, RequestContext& requestContext)
       at System.ServiceModel.Dispatcher.ErrorHandlingReceiver.EndTryReceive(IAsyncResult result, RequestContext& requestContext)
    [2015-08-28 11:20:34.357] Process:Microsoft.Crm.Sandbox.WorkerProcess |Organization:00000000-0000-0000-0000-000000000000 |Thread:   11 |Category: Sandbox |User: 00000000-0000-0000-0000-000000000000 |Level: Warning |ReqId: 00000000-0000-0000-0000-000000000000 | SandboxErrorHandler.HandleError  ilOffset = 0xE
     at SandboxErrorHandler.HandleError(Exception error)  ilOffset = 0xE
     at ErrorBehavior.HandleErrorCommon(Exception error, ErrorHandlerFaultInfo& faultInfo)  ilOffset = 0x38
     at ChannelDispatcher.HandleError(Exception error, ErrorHandlerFaultInfo& faultInfo)  ilOffset = 0x35
     at ErrorHandlingReceiver.Close()  ilOffset = 0xB
     at ChannelHandler.HandleReceiveComplete(RequestContext context)  ilOffset = 0x5D
     at ChannelHandler.EndTryReceive(IAsyncResult result, RequestContext& requestContext)  ilOffset = 0x3A
     at ChannelHandler.AsyncMessagePump(IAsyncResult result)  ilOffset = 0x2E
     at AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)  ilOffset = 0x0
     at AsyncResult.Complete(Boolean completedSynchronously)  ilOffset = 0xC2
     at TryReceiveAsyncResult.OnReceive(IAsyncResult result)  ilOffset = 0x4B
     at AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)  ilOffset = 0x0
     at AsyncResult.Complete(Boolean completedSynchronously)  ilOffset = 0xC2
     at ReceiveAsyncResult.OnReceiveComplete(Object state)  ilOffset = 0x2E
     at SessionConnectionReader.OnAsyncReadComplete(Object state)  ilOffset = 0xBF
     at AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)  ilOffset = 0x0
     at LazyAsyncResult.Complete(IntPtr userToken)  ilOffset = 0x3E
     at NegotiateStream.ProcessFrameBody(Int32 readBytes, Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)  ilOffset = 0x70
     at NegotiateStream.ReadCallback(AsyncProtocolRequest asyncRequest)  ilOffset = 0x68
     at FixedSizeReader.CheckCompletionBeforeNextRead(Int32 bytes)  ilOffset = 0x5D
     at FixedSizeReader.ReadCallback(IAsyncResult transportResult)  ilOffset = 0x29
     at AsyncResult.Complete(Boolean completedSynchronously)  ilOffset = 0xC2
     at IOAsyncResult.OnAsyncIOComplete(Object state)  ilOffset = 0x29
     at PipeConnection.OnAsyncReadComplete(Boolean haveResult, Int32 error, Int32 numBytes)  ilOffset = 0x143
     at OverlappedContext.CompleteCallback(UInt32 error, UInt32 numBytes, NativeOverlapped* nativeOverlapped)  ilOffset = 0x4C
     at IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)  ilOffset = 0x5
     at _IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)  ilOffset = 0x3C
    >System.ServiceModel.CommunicationObjectFaultedException: The communication object, System.ServiceModel.Channels.ServerSessionPreambleConnectionReader+ServerFramingDuplexSessionChannel, cannot be used for communication because it is in the Faulted state.
       at System.ServiceModel.Channels.CommunicationObject.Close(TimeSpan timeout)
       at System.ServiceModel.Dispatcher.ErrorHandlingReceiver.Close()

    • Edited by Alan.M Friday, August 28, 2015 7:46 PM
    Friday, August 28, 2015 5:46 PM
  • I've opened a support case with Microsoft and will post the findings here.
    Monday, August 31, 2015 3:43 PM
  • Alan, I believe support will be reaching out to you but for the benefit for others that might run into similar issues I'm adding some info from the support engineer that might be useful for others with a similar issue

    Here are the results of another scenario where we have seen this issue and fixed it through registry and web.config modifications.

    First, the web.config file for CRM will need 2 values changed:

    1. MaxRequestLength = 100MB

    2. maxAllowedContentLength = 100MB

    Next, the following Registry settings will need to be changed in the MSCRM hive of the Sandbox Server(s)

    1. [HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\MSCRM]

      1. "SandboxWcfHostMaxReceivedMessageSize"=dword: 616690c

      2. "SandboxWcfWorkerClientMaxReceivedMessageSize"=dword: 616690c

      3. "SandboxWcfWorkerMaxReceivedMessageSize"=dword: 616690c

      4. "SandboxWcfClientMaxReceivedMessageSize"=dword: 616690c

      5. "SandboxWcfSdkClientMaxReceivedMessageSize"=dword: 616690c

      6. "SandboxWcfSdkListenerMaxReceivedMessageSize"=dword: 616690c

    These values are tested  to allow for 32MB attachments as the application maximum would allow.  The values to allow every possible 32MB file to be uploaded could be higher than this as the compression algorithms can have an impact here. 


    .NET MVP / Microsoft Regional Director / http://blog.davidyack.com / http://crm.davidyack.com

    Monday, August 31, 2015 8:28 PM