locked
Transfers not working because call throttle is reached RRS feed

  • Question

  • We just received the following error on our speech server:

     

    Event Type: Warning
    Event Source: Office Communications Server 2007 Speech Server
    Event Category: Telephony Application Host
    Event ID: 32771
    Date:  7/19/2007
    Time:  2:12:52 PM
    User:  N/A
    Computer: PEPE
    Description:
    The requested operation 'OpenAsync()' on the Telephony Session failed for the following reason: 'The session cannot be created because the configured maximum number of outbound sessions has been created.'.  
     
    Further trace information for support personnel follows:
     
    Microsoft.SpeechServer.SpeechEngineServices.MaxConfiguredSessionsException: The session cannot be created because the configured maximum number of outbound sessions has been created.

    Server stack trace:
       at Microsoft.SpeechServer.SessionManager.CheckNumberOfSessions(SesContext context, Boolean inbound)
       at Microsoft.SpeechServer.SessionManager.CreateSession(SesContext context, IBrokerSessionListener listener, Boolean inbound, Boolean applyRateThrottling, IRemotingEventBatchReceiver sesProxyReceiver, Guid sesProxyReceiverId, IRemotingEventBatchReceiver& sesMainReceiver, Guid& sesMainReceiverId, Boolean proxyInDebugger)
       at Microsoft.SpeechServer.Broker.CreateSession(SesContext sesContext, IBrokerSessionListener listener, Boolean inbound, Boolean applyRateThrottling, IList`1& supportedVoices, IRemotingEventBatchReceiver sesProxyReceiver, Guid sesProxyReceiverId, IRemotingEventBatchReceiver& sesMainReceiver, Guid& sesMainReceiverId, Boolean proxyInDebugger)
       at Microsoft.SpeechServer.AuthenticatedBrokerSessionManager.CreateSession(SesContext sesContext, IBrokerSessionListener listener, Boolean inbound, Boolean applyRateThrottling, IList`1& supportedVoices, IRemotingEventBatchReceiver sesProxyReceiver, Guid sesProxyReceiverId, IRemotingEventBatchReceiver& sesMainReceiver, Guid& sesMainReceiverId, Boolean proxyInDebugger)
       at System.Runtime.Remoting.Messaging.StackBuilderSink._PrivateProcessMessage(IntPtr md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
       at System.Runtime.Remoting.Messaging.StackBuilderSink.PrivateProcessMessage(RuntimeMethodHandle md, Object[] args, Object server, Int32 methodPtr, Boolean fExecuteInContext, Object[]& outArgs)
       at System.Runtime.Remoting.Messaging.StackBuilderSink.SyncProcessMessage(IMessage msg, Int32 methodPtr, Boolean fExecuteInContext)

    Exception rethrown at [0]:
       at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
       at Microsoft.SpeechTransport.CreateSessionDelegate.EndInvoke(IList`1& supportedVoices, IRemotingEventBatchReceiver& sesMainReceiver, Guid& sesMainReceiverId, IAsyncResult result)
       at Microsoft.SpeechTransport.SessionInfo.CreateSessionComplete(IAsyncResult ar)

    For more information, see Help and Support Center at http://go.microsoft.com/fwlink/events.asp.

     

    We had the outbound call throttle set to 25 and we submitted a bunch of calls to the MSMQ.  This warning occured when a transfer from one of the calls was attempted.  It failed because there were already 25 calls in progress.

     

    My question is, is there any way to have MSS ignore the outbound call throttle for transfers?  As it is now, if we raise the throttle, MSS will just start making more calls and it will still be impossible to do transfers.  The only other solution I see is for us to throttle the amount of calls we submit to the MSMQ at a lower number than the MSS throttle. 

     

    Thursday, July 19, 2007 2:38 PM

Answers

  • Hi Dan,

     

    I work with John (sectrean). 

     

    We'll look into the solution you suggested, but I don't think it's going to work out.  We wrote our own SIP proxy/b2bua to go between MSS and our terminators, who don't speak TCP or REFER.  We're hairpinning the calls ourselves, and the flow of operations between MSS and the proxy is a bit delicate.  We'd probably need to re-write the transfer handling section of the proxy, and that's a real can of worms to open.

     

    I got a real kick out of this:

     

     Dan Herron [MSFT] wrote:

     

    As I said, I didn't try this myself, and you'll probably want to print out the various SIP RFCs (on heavy stock, so that they can withstand repeated readings and fits of anger) and make sure you have access to the SIP logs. And in any case, this solution seems rather tricky. But I offer it in case it works for you.

     

    Dan

     

    My printouts of the SIP/SDP/RTP specs are well-dog-eared and cursed at.  My opinion is that the specs are just a bit too complex to expect any two development teams to implement them the same, thus all the interoperability problems when doing anything beyond the basics.

     

     

    I whipped up a quick app to extract applications instances from ETL files (after you convert them to .txt).  I should modify it to find the application instance by phone number, but I need more coffee first.  Once I figure out which calls were involved with the transfer exception, we'll email them to you. 

     

    Other people might find the app useful, so here it is:

     

    using System;

    using System.Collections.Generic;

    using System.IO;

    using System.Text;

    namespace ExtractMssCall

    {

    class Program

    {

    static string applicationInstance;

    static string fileName;

    static string outFile;

    static void Main(string[] args)

    {

    if (args.Length != 3)

    {

    Console.WriteLine("Usage: ExtractMssCall.exe traceFile outFile applicationInstance");

    return;

    }

    fileName = args[0];

    outFile = args[1];

    applicationInstance = args[2];

    using (StreamReader sr = new StreamReader(fileName))

    {

    using (StreamWriter sw = new StreamWriter(outFile))

    {

    StringBuilder currentTrace = new StringBuilder();

    while (true)

    {

    string line = sr.ReadLine();

    if (line == null)

    {

    break;

    }

    currentTrace.Append(line + "\r\n");

    if (line.StartsWith("}"))

    {

    string trace = currentTrace.ToString();

    if (ProcessTrace(trace))

    {

    sw.WriteLine(trace);

    }

    currentTrace = new StringBuilder();

    }

    }

    }

    }

    }

    static bool ProcessTrace(string trace)

    {

    using (StringReader sr = new StringReader(trace))

    {

    while (true)

    {

    string line = sr.ReadLine();

    if (line == null)

    {

    break;

    }

    if (line.IndexOf("ApplicationInstance") > -1)

    {

    string[] tokens = line.Split(' ');

    foreach (string token in tokens)

    {

    if (token == applicationInstance)

    {

    return true;

    }

    }

    }

    }

    }

    return false;

    }

    }

    }

     

    Eric Z. Beard

    www.loopfx.com

     

    Friday, July 20, 2007 1:41 PM

All replies

  • Also, the OpenCompleted event was never raised.  As a result, the application just hung. 

     

    I would have expected the OpenCompleted event to have been raised with an appropriate exception. Then, at least, the original call could have been reconnected and a message could have been given to the user. 

    Thursday, July 19, 2007 2:45 PM
  • Hi,

     

    A few questions:

     

    Can you use a blind transfer, instead?

    1. Did you wire up to the OpenComplete on the first session or the second one? The event should be raised on the second (new) session.
    2. Also, why was the original caller not still connected? I assume you are doing this:
    • get a new TelephonySession (from MSMQ)
    • place the call, and wait for it to connect
    • interact with the called party
    • create the second TelephonySession
    • place the call, and wait for success/failure
    • if success, call TransferAsync()
    • if failure, apologize to the first called party

    I believe that when the second call fails to connect, the first call should still be connected.

     

    But to answer your question, there is no easy way to set aside capacity reserved exclusively for supervised transfer sessions. To do that, your best bet will be to (1) monitor the MSS perf counters to see how many sessions are active; (2) decide how many primary sessions you want to allow; and (3) allow items to trickle into MSMQ only fast enough to keep yourself at that limit.

     

    Dan

    Thursday, July 19, 2007 6:25 PM
  • Blind transfer is not really an option for us. 

     

    We are making the first call with the regular MakeCallActivity.  Then in a custom SupervisedTransferActivity we are putting the first session on hold and attempting to connect a second call. 

     

    We are wiring up the OpenComplete event on the second session.  Normally, this event fires and if the second call is connected, we proceed with the transfer, or if it didn't connect, we take the original call off hold and play a message.  This has all been tested and works. 

     

    However, in this particular case, the OpenComplete event never fired.  The original call was still connected, but it was still on hold.  Since the event never fired, there was no way to take the original call off of hold.

     

     

    Thursday, July 19, 2007 6:42 PM
  • Could you send me MSS logs for this failure?

     

    One other way you might handle this (I have not attempted to do this, so it might not actually work, and it is clearly tricky):

    • when you want to make the second call, post a message to your own MSMQ (the same one that the current message arrived on)
    • Set the priority on the message so that it's higher than the normal messages. Make the body something you can recognize (i.e., distinct from the messages used for the normal calls), and include in the body of this message the current values for the following (make these query params so that in the new application instance you will easily be able to read them)
      • TelephonySession.Id
      • TelephonySession.CallInfo.FromTag
      • TelephonySession.CallInfo.ToTag
      • TelephonySession.CallInfo.CalledParty.Uri
    • In your app, handle those two types of MSMQ messages separately. Handle these new messages with code that calls OpenAsync, waits for OpenCompleted, and then calls TransferAsync(SipUri address). I.e., you are seeming to make a blind transfer on the second session.

    For the 'address', pass a new SipUri that is composed like this (making sure to escape the Replaces header)

    Code Snippet

    string referToUri = address.ToString(); // address you passed from the first session

    referToUri += String.Format(CultureInfo.InvariantCulture,
        "?Replaces={0}%3Bto-tag%3D{1}%3Bfrom-tag%3D{2}",
        _otherCallId,// the CallID of the other session
        _otherCallToTag, // the ToTag of the other session
        _otherCallFromTag); // the FromTag of the other session

     

    If I'm right, this would cause you to complete a supervised transfer between the two TelephonySessions. (and you would detect this in the first app instance by getting a TelephonySession.Closed event).

     

    As I said, I didn't try this myself, and you'll probably want to print out the various SIP RFCs (on heavy stock, so that they can withstand repeated readings and fits of anger) and make sure you have access to the SIP logs. And in any case, this solution seems rather tricky. But I offer it in case it works for you.

     

    Dan

    Thursday, July 19, 2007 8:38 PM
  • The log file for this hour is 16MB zipped and it's getting rejected. 

     

    Is there an easy way to extract the data for one call from the ETL files?  If not, I guess I can split up the zip file.

    Thursday, July 19, 2007 9:54 PM
  • Make sure that you have not in this case enabled any of the 'audio' settings in logging. And make sure to start a new log file immediately before and after the repro, to keep the size as small as possible. Unless you need to run for a long time to repro, that should keep the size manageable.

     

    We do not ship a tool that allows you to extract a single call from the logs. You can open the logs in the MSS tuning tools, but there is no way for you to send me the logs for just one call.

     

    Thursday, July 19, 2007 11:20 PM
  • Hi Dan,

     

    I work with John (sectrean). 

     

    We'll look into the solution you suggested, but I don't think it's going to work out.  We wrote our own SIP proxy/b2bua to go between MSS and our terminators, who don't speak TCP or REFER.  We're hairpinning the calls ourselves, and the flow of operations between MSS and the proxy is a bit delicate.  We'd probably need to re-write the transfer handling section of the proxy, and that's a real can of worms to open.

     

    I got a real kick out of this:

     

     Dan Herron [MSFT] wrote:

     

    As I said, I didn't try this myself, and you'll probably want to print out the various SIP RFCs (on heavy stock, so that they can withstand repeated readings and fits of anger) and make sure you have access to the SIP logs. And in any case, this solution seems rather tricky. But I offer it in case it works for you.

     

    Dan

     

    My printouts of the SIP/SDP/RTP specs are well-dog-eared and cursed at.  My opinion is that the specs are just a bit too complex to expect any two development teams to implement them the same, thus all the interoperability problems when doing anything beyond the basics.

     

     

    I whipped up a quick app to extract applications instances from ETL files (after you convert them to .txt).  I should modify it to find the application instance by phone number, but I need more coffee first.  Once I figure out which calls were involved with the transfer exception, we'll email them to you. 

     

    Other people might find the app useful, so here it is:

     

    using System;

    using System.Collections.Generic;

    using System.IO;

    using System.Text;

    namespace ExtractMssCall

    {

    class Program

    {

    static string applicationInstance;

    static string fileName;

    static string outFile;

    static void Main(string[] args)

    {

    if (args.Length != 3)

    {

    Console.WriteLine("Usage: ExtractMssCall.exe traceFile outFile applicationInstance");

    return;

    }

    fileName = args[0];

    outFile = args[1];

    applicationInstance = args[2];

    using (StreamReader sr = new StreamReader(fileName))

    {

    using (StreamWriter sw = new StreamWriter(outFile))

    {

    StringBuilder currentTrace = new StringBuilder();

    while (true)

    {

    string line = sr.ReadLine();

    if (line == null)

    {

    break;

    }

    currentTrace.Append(line + "\r\n");

    if (line.StartsWith("}"))

    {

    string trace = currentTrace.ToString();

    if (ProcessTrace(trace))

    {

    sw.WriteLine(trace);

    }

    currentTrace = new StringBuilder();

    }

    }

    }

    }

    }

    static bool ProcessTrace(string trace)

    {

    using (StringReader sr = new StringReader(trace))

    {

    while (true)

    {

    string line = sr.ReadLine();

    if (line == null)

    {

    break;

    }

    if (line.IndexOf("ApplicationInstance") > -1)

    {

    string[] tokens = line.Split(' ');

    foreach (string token in tokens)

    {

    if (token == applicationInstance)

    {

    return true;

    }

    }

    }

    }

    }

    return false;

    }

    }

    }

     

    Eric Z. Beard

    www.loopfx.com

     

    Friday, July 20, 2007 1:41 PM