locked
Inbound/Outbound PSTN calls stopped working in OCS 2007 R2! RRS feed

  • Question

  • Hi All,

    I just setup my OCS 2007 R2 environment. I have OCS 2007 R2 Front End server (OCSFNTED.domaincorp.com), Mediation server (OCSMEDTN.domaincorp.com) that is connected to an AudioCodes MG1000 gateway (MEDIANT1.domaincorp.com) and connected to an E1R2, and a Consolidated Edge server (OCSEDGE).

    I got everything working fine—for about 2 weeks now. I have Enterprise Voice enabled users. All inbound/outbound call scenarios to PSTN were working until now. When I checked my OC 2007 R2 event logs in my client laptop, I got this error...

    ---------------------------------------------------------------------------------------------------------------------------------

    A SIP request made by Communicator failed in an unexpected manner (status code 80ef0194). More information is contained in the following technical data:
     
     RequestUri:   sip:<Tel URI>@domain.com.ph;user=phone
    From:         sip:surname.firstname@domain.com.ph;tag=c09e50cd21
    To:           sip: <Tel URI>@domain.com.ph;user=phone;tag=663efd74d
    Call-ID:      4dd38a6132c3491a9737894f2c1d02e4
    Content-type: application/sdp;call-type=audiovideo

    (null)

    Response Data:

    101  Progress Report
    ms-diagnostics:  12006;reason="Trying next hop";source="OCSFNTED.domaincorp.com";PhoneUsage="CN={0E7B89F2-39A0-4B50-867E-CC987695574E},CN=Phone Route Usages,CN=RTC Service,CN=Services,CN=Configuration,DC=domaincorp,DC=com";PhoneRoute="Makati-T1 Local";Gateway="OCSMEDTN.domaincorp.com:5061";appName="OutboundRouting"


    183  Session Progress


    404  Not Found
    Ms-diagnostics:  10404;source="OCSMEDTN.domaincorp.com";reason="Gateway returned a SIP failure code";component="MediationServer";SipResponseCode="404";SipResponseText="Not Found";GatewayFqdn="MEDIANT1.domaincorp.com"

     
     Resolution:
     If this error continues to occur, please contact your network administrator. The network administrator can use a tool like winerror.exe from the Windows Resource Kit or lcserror.exe from the Office Communications Server Resource Kit in order to interpret any error codes listed above.

    ---------------------------------------------------------------------------------------------------------------------------------

    I checked my front end server, and i got an error... its event id 46009...

    --------------------------------------------------------------------------------------------------------------------------------

    Log Name:      Office Communications Server
    Source:        OCS Outbound Routing
    Date:          3/6/2009 2:46:37 PM
    Event ID:      46009
    Task Category: (1038)
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      OCSFNTED.domaincorp.com
    Description:
    An attempt to route to a gateway failed.

    Could not route to Gateway OCSMEDTN.domaincorp.com:5061, the attempt failed with response code: Timeout.
    Failure occurrences: 2, since 3/6/2009 2:46:07 PM.
    Cause: A gateway failed to respond to a request within allotted time or was unable to route the request due to some error.
    Resolution:
    Check whether the specified gateway is up and is properly configured.

    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="OCS Outbound Routing" />
        <EventID Qualifiers="50190">46009</EventID>
        <Level>2</Level>
        <Task>1038</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2009-03-06T06:46:37.000Z" />
        <EventRecordID>108783</EventRecordID>
        <Channel>Office Communications Server</Channel>
        <Computer>OCSFNTED.domaincorp.com</Computer>
        <Security />
      </System>
      <EventData>
        <Data>OCSMEDTN.domaincorp.com:5061</Data>
        <Data>Timeout</Data>
        <Data>2</Data>
        <Data>3/6/2009 2:46:07 PM</Data>
      </EventData>
    </Event>

    ----------------------------------------------------------------------------------------------------------------------------------

    Then, I checked my mediation server, and I saw a bunch of errors. Its either Event ID 25045 or 25046.

    Event ID 25045 is...

    ---------------------------------------------------------------------------------------------------------------------------------

    Log Name:      Office Communications Server
    Source:        OCS Mediation Server
    Date:          3/5/2009 2:38:15 PM
    Event ID:      25045
    Task Category: (1030)
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      OCSMEDTN.domaincorp.com
    Description:
    The Quality-Metric Agent cannot be contacted. Mediation server will keep trying; however additional failures will not be logged.

    QoE Agent: OCSFNTED.domaincorp.com:5069
    GRUU: sip:OCSFNTED.domaincorp.com@domaincorp.com;gruu;opaque=srvr:QoS:YJEl8-MQV0q98eBf2l2QLAAA

    Cause: Either the Quality-Metric Agent is not running or unreachable.
    Resolution:
    Verify that Quality-Metric Agent is reachable from the computer running Office Communications Server.

    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="OCS Mediation Server" />
        <EventID Qualifiers="50182">25045</EventID>
        <Level>2</Level>
        <Task>1030</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2009-03-05T06:38:15.000Z" />
        <EventRecordID>751</EventRecordID>
        <Channel>Office Communications Server</Channel>
        <Computer>OCSMEDTN.domaincorp.com</Computer>
        <Security />
      </System>
      <EventData>
        <Data>OCSFNTED.domaincorp.com:5069</Data>
        <Data>sip:OCSFNTED.domaincorp.com@domaincorp.com;gruu;opaque=srvr:QoS:YJEl8-MQV0q98eBf2l2QLAAA</Data>
      </EventData>
    </Event>

    -------------------------------------------------------------------------------------------------------------------------------

    Event ID 25046 is...

    ----------------------------------------------------------------------------------------------------------------------------

    Log Name:      Office Communications Server
    Source:        OCS Mediation Server
    Date:          3/5/2009 2:38:15 PM
    Event ID:      25046
    Task Category: (1030)
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      OCSMEDTN.domaincorp.com
    Description:
    The Quality-Metric Agent returned invalid response. The Quality metric reports are not sent.

    Response Code: 504
    Response Text: Server time-out
    QoE Agent: OCSFNTED.domaincorp.com:5069
    GRUU: sip:OCSFNTED.domaincorp.com@domaincorp.com;gruu;opaque=srvr:QoS:YJEl8-MQV0q98eBf2l2QLAAA
    Diagnostic Code: 1007
    Diagnostic Source: OCSFNTED.domaincorp.com
    Diagnostic Reason: Temporarily cannot route

    Cause: Either the Quality-Metric Agent is not running or unreachable.
    Resolution:
    Verify that Quality-Metric Agent is reachable from the computer running Office Communications Server. The response text and Diagnostic information may also indicate a configuration issue.
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="OCS Mediation Server" />
        <EventID Qualifiers="50182">25046</EventID>
        <Level>2</Level>
        <Task>1030</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2009-03-05T06:38:15.000Z" />
        <EventRecordID>750</EventRecordID>
        <Channel>Office Communications Server</Channel>
        <Computer>OCSMEDTN.domaincorp.com</Computer>
        <Security />
      </System>
      <EventData>
        <Data>504</Data>
        <Data>Server time-out</Data>
        <Data>OCSFNTED.domaincorp.com:5069</Data>
        <Data>sip:OCSFNTED.domaincorp.com@domaincorp.com;gruu;opaque=srvr:QoS:YJEl8-MQV0q98eBf2l2QLAAA</Data>
        <Data>1007</Data>
        <Data>OCSFNTED.domaincorp.com</Data>
        <Data>Temporarily cannot route</Data>
      </EventData>
    </Event>

    --------------------------------------------------------------------------------------------------------------------------------

    Then, I checked the logs of my AudioCodes MG1000 gateway. First, I enabled Message Logging. When I do outbound calls, I get this message...

    ---------------------------------------------------------------------------------------------------------------------------------

    Log is Activated

    1d:1h:5m:34s ( lgr_TrnkGrp)(85 ) !! [ERROR] #2:TrunkGroup::AllocateEndPoint- Can't find EndPoint for phone number 028581000

    1d:1h:5m:34s ( lgr_psbrdif)(86 ) !! [ERROR] MotherBoard::GetEndPoint- Can't find EndPoint for Dest:028581000 Source:027939553 SourceIp:1cba460

    1d:1h:5m:34s ( lgr_call)(87 ) !! [ERROR] Call::GetEndPoint- Can't find endpoint for phone number 028581000

    1d:1h:5m:34s |CALL_END |-1 |0 |-1 |-1 |-1 |-1 |UNKNOWN |IP |0.0.0.0 |0.0.0.0 |0 |0 |0<AreaCode + 7 Digit # of Calling Party > |+<CountryCode + AreaCode +7 Digit # of Calling Party> |0 |0 |0<AreaCode + 7 Digit # of Called Party> |+<CountryCode + AreaCode + 7 Digit # of Called Party> |0 |N/A |255 |0.0.0.0 |0 |TEL |GWAPP_NO_ROUTE_TO_DESTINATION |0 |-1 |-1 |-1 |b8ed87f8-d68c-4173-a497-d21403d9b075 | | | |-1 |-1 |0 |0 |-1 |0 |0 | |0


    1d:1h:5m:34s ( lgr_call)(89 ) ?? [WARNING] Call::PassEvToOtherSide m_EndPoint is NULL!!

    1d:1h:5m:34s task GWAP, Un nested AcLockStart

    1d:1h:5m:34s task GWAP, Mismatch preemption mode

    ----------------------------------------------------------------------------------------------------------------------------------

    When I do inbound call from a PSTN user, no logs are generated. In the AudioCodes gateway. I already restarted all my servers and the services were all running.

    What could have caused this sudden problem in my inbound/outbound PSTN calls? Is the error OCS server related? What is it? 

     

    • Edited by Vince Q. _ Monday, June 15, 2009 11:17 AM
    Friday, March 6, 2009 10:07 AM

Answers

  • So, there are two things going on here, and both may have different root causes, or there may be some overlap. But for now, think of the Quality Metric server errors separately from your call failures.

    I don't have time tonight to address the Quality Metric errors, your call failures seem pretty straight forward. Your AudioCodes Mediant gateway is not connecting to the PSTN any longer.

    • Your outbound calls fail because the gateway doesn't have a connection.
    • Your inbound calls do not generate any logs because, again, your gateway doesn't have a connection. Calls from the PSTN are not even reaching the gateway.

    Some questions: Are you the only administrator? When was the last modification to the gateway performed? Does it also serve as a gateway to Exchange UM, or any other solutions?

    Suggestions: 

    Start by checking the Mediant's connection and status: 

    • Check the AudioCodes configuration interface/front panel for errors/connectivity information.
      • Does it show connectivity up on the connection to the PSTN/PBX?
      • If someone made recent changes to the AudioCodes config, try exporting your current config, saving that somewhere secure, and reloading your old config file (which you hopefully have backed up).
    • Check Physical connections.
      • Make sure your T1/E1/whatever is physically connected to the correct ports.
      • Try disconnecting and reconnecting it. 
    • If your gateway is connected directly to a carrier via T1 or whatever, check with whoever owns the other end of it.
    • If your gateway is connected to a PBX, check with the PBX administrator about the status of the connection.
    • If the gateway has connections for analog handsets, is it configured to enable one to dial to the PSTN? if so, is that working?
    • Try power cycling the gateway. I recommend you back up the current config before doing so, especially in the case where someone else may have administrative access, and may have made changes to the gateway.

    Duncan Blake, Enterprise Voice Architect, Unify2
    • Proposed as answer by Duncan Blake Thursday, March 19, 2009 3:19 AM
    • Marked as answer by Vince Q. _ Tuesday, October 20, 2009 9:34 AM
    Tuesday, March 10, 2009 12:59 AM

All replies

  • Looks like a certificate problem to me. Check your certs and also make sure that your Mediant and Mediation are both either talking non-secure(5060) or secure(5061).

    Monday, March 9, 2009 2:50 AM
  • Hey Mark,

    Thanks for replying!

    Actually, everything was working for a month now. Nothing was changed or configrued in the working setup. Both the gateway and Mediation are configured for either 5060 or 5061. Then, just last friday PSTN calls both inbound/outbound gets disconnected. Then, I got the "OCS Outbound Routing" error in the Front End server. Then the gateway does not capture any logs for inbound calls from PSTN.



    vinceq
    • Edited by Vince Q. _ Monday, June 15, 2009 11:16 AM
    Monday, March 9, 2009 3:57 AM
  • So, there are two things going on here, and both may have different root causes, or there may be some overlap. But for now, think of the Quality Metric server errors separately from your call failures.

    I don't have time tonight to address the Quality Metric errors, your call failures seem pretty straight forward. Your AudioCodes Mediant gateway is not connecting to the PSTN any longer.

    • Your outbound calls fail because the gateway doesn't have a connection.
    • Your inbound calls do not generate any logs because, again, your gateway doesn't have a connection. Calls from the PSTN are not even reaching the gateway.

    Some questions: Are you the only administrator? When was the last modification to the gateway performed? Does it also serve as a gateway to Exchange UM, or any other solutions?

    Suggestions: 

    Start by checking the Mediant's connection and status: 

    • Check the AudioCodes configuration interface/front panel for errors/connectivity information.
      • Does it show connectivity up on the connection to the PSTN/PBX?
      • If someone made recent changes to the AudioCodes config, try exporting your current config, saving that somewhere secure, and reloading your old config file (which you hopefully have backed up).
    • Check Physical connections.
      • Make sure your T1/E1/whatever is physically connected to the correct ports.
      • Try disconnecting and reconnecting it. 
    • If your gateway is connected directly to a carrier via T1 or whatever, check with whoever owns the other end of it.
    • If your gateway is connected to a PBX, check with the PBX administrator about the status of the connection.
    • If the gateway has connections for analog handsets, is it configured to enable one to dial to the PSTN? if so, is that working?
    • Try power cycling the gateway. I recommend you back up the current config before doing so, especially in the case where someone else may have administrative access, and may have made changes to the gateway.

    Duncan Blake, Enterprise Voice Architect, Unify2
    • Proposed as answer by Duncan Blake Thursday, March 19, 2009 3:19 AM
    • Marked as answer by Vince Q. _ Tuesday, October 20, 2009 9:34 AM
    Tuesday, March 10, 2009 12:59 AM
  • Hey Duncan,

    Thanks for replying!

    I did realized the Quality Metric server had nothing to do with the call failures. :)

    Access wise, I assigned a couple of people (including myself) in the RTCUniversalServerAdmins group. I already double checked that, and nothing seem changed in the server configurations. Same with the AudioCodes gateway.

    The calls, IP-to-Tel, routes correctly and it shows event logs in the gateway. But it has a can't find destination error or something like that. So I did checked my AudioCodes gateway and saw the LOS/LOF alarm or warning on the E1 interface. Configuration wise, nothing was changed in both the OCS servers and gateway since it started working, so I'm having the E1R2 checked by the Telco provider. 

    Anyways, thanks again for the suggestions! 



    Tuesday, March 10, 2009 3:26 AM
  • It's actually working now. The E1R2 was the problem. Thank you for the detailed suggestions! :)
    vinceq
    Thursday, March 19, 2009 3:15 AM
  • Glad to hear it!


    Duncan Blake, Enterprise Voice Architect, Unify2
    Thursday, March 19, 2009 3:19 AM