none
Running custom WCF on IaaS-Generated Azure cluster from local client - Getting time out request on EndRequest RRS feed

  • Question

  • I've run the test HelloWorld program from a local client while the cluster was on Azure and was able to get that to run successfully.  I am now trying to switch off to my own WCF service and am getting time-out errors on Broker.EndRequest().

    I have used the (int32) version of that and flush, set as high as 600 000 ms and the time out persists.  I have also updated my config file to more closely resemble that of the EchoService example (there were XML nodes not included in my service.)  I am unable to get the function calls to commit and the cluster to actually run on the inputs.

    I have included the full error below.  Numerictest is a simple test function in my service used for testing purposes.  it is simply fed a number and returns true if it's an integer, false otherwise (the request inputs are generated in a for loop, 10 inputs, each input generated as i * 0.25.  Locally I have used 400 inputs with no issue); all of two lines and incredibly basic.

    System.TimeoutException was unhandled
      HResult=-2146233083
      Message=BrokerPersistQueue flush timeout for waiting for the incoming requests within 60000 milliseconds for client[clientId=].
      Source=Microsoft.Hpc.Scheduler.Session
      StackTrace:
           at Microsoft.Hpc.Scheduler.Session.BrokerClient`1.Flush(Int32 timeoutMilliseconds, Boolean endOfMessage)
           at Microsoft.Hpc.Scheduler.Session.BrokerClient`1.EndRequests()
           at LocalSimpleWCFClient.Module1.NumericTest()
           at LocalSimpleWCFClient.Module1.Main(String[] args)
           at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
           at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
           at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
           at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
           at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
           at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
           at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
           at System.Threading.ThreadHelper.ThreadStart()
      InnerException:

    Monday, January 18, 2016 10:47 PM

Answers

  • Hi KWilliams1,


    Please check the following:

    1. Was the SOA session job created successfully and in running state? What's the number of incoming requests?

    2. Are you using Azure storage by setting info.UseAzureQueue = true? Could you try info.UseAzureQueue = false to use https binding directly for sending the requests?

    3. Double check if info.Username is set with format [Domain]\[Username] instead of only [Username].

    If no further findings, please collect the sesson and broker logs and send to me (yutongs@microsoft.com) for further investigation.

    a) To collect Session API logs, created a c:\TEMP\ folder and then add the following under the <configuration> section in app.config under your client project or the <appClient>.exe.config file:
     <system.diagnostics>
     <trace autoflush="true"/>
     <sharedlisteners>
     <add name="xml" type="System.Diagnostics.XmlWriterTraceListener" initializedata="c:\TEMP\session.svclog"/>
     </sharedlisteners>
     <sources>
     <source name="SOA Session API" switchvalue="All">
     <listeners>
     <remove name="Default"/>
     <add name="xml"/>
     </listeners>
     </source>
     </sources>
     </system.diagnostics>

    b) To collect broker logs, add an attribute PerSessionLogging="1" for the shared listener “SoaListener” in HpcBrokerWorker.exe.config under %CCP_HOME%Bin on the head node/broker node and then restart the HpcBroker service. After that, when a SOA session with id [SessionId] finishes, there would be a file named HpcBrokerWorker_[LogIdentifier]_[SessionId] under %CCP_DATA%LogFiles\SOA folder on the head node/broker node. All the broker log files for this SOA session are named like HpcBrokerWorker_[LogIdentifier]_*.bin, they are by default 1MB files.

    Then repro the issue, after the session job is finished or canceled collect the session log file c:\TEMP\session.svclog on the client machine and the broker log files HpcBrokerWorker_[LogIdentifier]_*.bin on the head node/broker node.

    BR,

    Yutong Sun


    • Marked as answer by KWilliams1 Tuesday, January 19, 2016 8:01 PM
    Tuesday, January 19, 2016 11:29 AM

All replies

  • Hi KWilliams1,


    Please check the following:

    1. Was the SOA session job created successfully and in running state? What's the number of incoming requests?

    2. Are you using Azure storage by setting info.UseAzureQueue = true? Could you try info.UseAzureQueue = false to use https binding directly for sending the requests?

    3. Double check if info.Username is set with format [Domain]\[Username] instead of only [Username].

    If no further findings, please collect the sesson and broker logs and send to me (yutongs@microsoft.com) for further investigation.

    a) To collect Session API logs, created a c:\TEMP\ folder and then add the following under the <configuration> section in app.config under your client project or the <appClient>.exe.config file:
     <system.diagnostics>
     <trace autoflush="true"/>
     <sharedlisteners>
     <add name="xml" type="System.Diagnostics.XmlWriterTraceListener" initializedata="c:\TEMP\session.svclog"/>
     </sharedlisteners>
     <sources>
     <source name="SOA Session API" switchvalue="All">
     <listeners>
     <remove name="Default"/>
     <add name="xml"/>
     </listeners>
     </source>
     </sources>
     </system.diagnostics>

    b) To collect broker logs, add an attribute PerSessionLogging="1" for the shared listener “SoaListener” in HpcBrokerWorker.exe.config under %CCP_HOME%Bin on the head node/broker node and then restart the HpcBroker service. After that, when a SOA session with id [SessionId] finishes, there would be a file named HpcBrokerWorker_[LogIdentifier]_[SessionId] under %CCP_DATA%LogFiles\SOA folder on the head node/broker node. All the broker log files for this SOA session are named like HpcBrokerWorker_[LogIdentifier]_*.bin, they are by default 1MB files.

    Then repro the issue, after the session job is finished or canceled collect the session log file c:\TEMP\session.svclog on the client machine and the broker log files HpcBrokerWorker_[LogIdentifier]_*.bin on the head node/broker node.

    BR,

    Yutong Sun


    • Marked as answer by KWilliams1 Tuesday, January 19, 2016 8:01 PM
    Tuesday, January 19, 2016 11:29 AM
  • It's was number 3.  Always the simplest things.  Thanks for the additional debugging info, which will come in handy moving forward.
    Tuesday, January 19, 2016 8:01 PM