RemotingException while submitting a job by code. RRS feed

  • Question

  • I have a c# windows app that communicates with a WCF service on the IIS that is on the Headnode of my cluster.

    This WCF service is used to submits 'jobs' on behalf of windows forms client. This way, I do not have to install/redistribute HPC client utilities with my windows forms app.

    When I submit a job from the WCF service , I get a remotingexception(User identity is not authorized to connect to this endpoint) on DurableSession.CreateSession()... The same user can submit jobs fine from windows forms client by creating a durablesession directly to the headnode. 

    {"Failed to get cluster property.

    The scheduler raised exception: System.Runtime.Remoting.RemotingException: An error occurred while processing the request on the server: System.Runtime.Remoting.RemotingException: User identity is not authorized to connect to this endpoint\r\n

    at System.Runtime.Remoting.Channels.Tcp.TcpServerChannel.AcceptSocketCallback(IAsyncResult ar)\r\n\r\nServer stack trace: \r\n

    at System.Runtime.Remoting.Channels.Tcp.TcpSocketHandler.ReadToEndOfHeaders(BaseTransportHeaders headers)\r\n

    at System.Runtime.Remoting.Channels.Tcp.TcpClientSocketHandler.ReadHeaders()\r\n

    at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)\r\n

    at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)\r\n\r\nException rethrown at [0]: \r\n at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)\r\n

    at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)\r\n at Microsoft.Hpc.Scheduler.Store.ISchedulerStoreInternal.GetClusterEnvironmentVariables(ConnectionToken token)\r\n at Microsoft.Hpc.Scheduler.Scheduler.get_EnvironmentVariables()\r\n

    at Microsoft.Hpc.Scheduler.Session.Internal.Common.ThreadSafeScheduler.get_EnvironmentVariables()\r\n at Microsoft.Hpc.Scheduler.Session.Internal.SessionLauncher.SessionLauncher.GetRegistrationRepo(String headnode, String callId)."}"

    Client side, I use 'Impersonation':

    m_svcMyServiceClient = new MyServiceClient();
    m_svcMyServiceClient.ClientCredentials.Windows.AllowedImpersonationLevel = System.Security.Principal.TokenImpersonationLevel.Impersonation

    Client config:

    <system.serviceModel> <bindings> <wsHttpBinding> <binding name="WSHttpBinding_IMyService" maxBufferPoolSize="2147483647" maxReceivedMessageSize="2147483647"/>

    <security mode="Message">
                <message clientCredentialType="Windows" negotiateClientCredential="false" establishSecurityContext="false"/>

    </wsHttpBinding> </bindings> <client> <endpoint address="http://<name>/foo/ProjectWorker.svc" binding="wsHttpBinding" bindingConfiguration="WSHttpBinding_IMyService" contract="Service.IMyService" name="WSHttpBinding_IMyService" />

              <servicePrincipalName value="HOST/<name>.bar.com"/>

    </client> </system.serviceModel> </configuration>

    IIS - 1) website on IIS on webserver has AnonymousAuth enabled (disabled everything else);

            2) apppool is running as 'ApplicationPoolIdentity'

            3) headnode (where iis is hosted) is trusted for delegation in AD

    WCF service config

    <bindings> <wsHttpBinding> <binding name="wsHttpBindingConfig" maxBufferPoolSize="2147483647" maxReceivedMessageSize="2147483647"> <readerQuotas maxDepth="32" maxStringContentLength="2147483647" maxArrayLength="2147483647" maxBytesPerRead="4096" maxNameTableCharCount="16384" /> <security mode="Message"> <message clientCredentialType="Windows" negotiateServiceCredential="false" establishSecurityContext="false"/> </security> </binding> </wsHttpBinding> </bindings>

    <service name="ProjectWorker" behaviorConfiguration="ProjectWorkerBehavior">
            <endpoint address="http://<name>/foo/ProjectWorker.svc"
              binding="wsHttpBinding" bindingConfiguration="wsHttpBindingConfig" contract="IMyService" >
                <servicePrincipalName value="HOST/<name>.bar.com"/>


    What am I possibly missing here?

    • Edited by SRIRAM R Tuesday, July 12, 2016 1:46 AM
    Monday, July 11, 2016 8:35 PM

All replies

  • I looked at https://social.microsoft.com/Forums/en-US/80d361aa-3ec8-4760-982d-8dee751ae346/how-to-pass-windows-credentials-to-hpc-2012-on-jobsubmit-api?forum=windowshpcsched

    and my server is set to delegation in AD.

    When I change the App pool identity of my WCF service to Local System or network service, the job is submitted as that local system or network user and an exception is found in the trace log (that is probably because network service or local system are not in HPC users group on head node):

    Authentication failure:  Please check your credentials and try again.

    System.ServiceModel.FaultException`1[[Microsoft.Hpc.Scheduler.Session.Internal.SessionFault, Microsoft.Hpc.Scheduler.Session, Version=, Culture=neutral, PublicKeyToken=31bf3856ad364e35]], System.ServiceModel, Version=, Culture=neutral, PublicKeyToken=b77a5c561934e089

    System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ProxyRpc&amp; rpc)
    System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result)
    System.ServiceModel.Channels.ServiceChannelProxy.InvokeEndService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
    System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
    System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData&amp; msgData, Int32 type)
    Microsoft.Hpc.Scheduler.Session.Internal.ISessionLauncher.EndAllocateDurable(Int32&amp; sessionid, String&amp; serviceVersion, IAsyncResult result)
    Microsoft.Hpc.Scheduler.Session.Internal.SessionLauncherClient.AllocateDurable(SessionStartInfoContract info, String endpointPrefix, Int32&amp; sessionid, String&amp; serviceVersion)
    Microsoft.Hpc.Scheduler.Session.Internal.ServiceJobProvider.AllocateResource(SessionStartInfoContract startInfo, Boolean durable, TimeSpan timeout, String[]&amp; eprs)
    Microsoft.Hpc.Scheduler.Session.Internal.OnPremiseSessionFactory.CreateSession(SessionStartInfo startInfo, Boolean durable, Int32 timeoutMilliseconds)
    Microsoft.Hpc.Scheduler.Session.DurableSession.CreateSession(SessionStartInfoBase startInfo)

    • Edited by SRIRAM R Tuesday, July 12, 2016 4:42 AM
    Tuesday, July 12, 2016 4:41 AM
  • Hi SRIRAM R,

    The problem could be that your service failed to impersonate the client credential. Please check you service code and configuration to see whether the impersonation in your service is achieved by the Imperative Model or the Declarative Model like the samle below,

    1. Imperative Model

    using (callerWindowsIdentity.Impersonate()) { // Create the SOA session here. }

    2. Declarative Model

    [OperationBehavior(Impersonation = ImpersonationOption.Required)]

    For details, you may refer to Delegation and Impersonation with WCF.


    Yutong Sun

    Tuesday, July 12, 2016 8:17 AM
  • I'm using declarative model. But still cannot get to impersonate/delegate the actual user to create the DurableSession()..

    Guess will cross post this on WCF forums as well.

    Tuesday, July 12, 2016 11:58 AM
  • Hi,

    Can someone from HPC team post a sample to get this to work (with settings used in IIS)? As I said earlier, IIS on same server as HPC head node and that head node has unconstrained delegation in AD.

    Too, the app pool under which the WCF service resides is running under 'Application Pool Identity'. All I am trying to do is get the remote user connect to my WCF service, which in turn, would create a durable session and submit requests to HPC head node (on the same server) using the user's AD account.

    Environment is Windows 2012 R2, IIS 8.5

    When I change the app pool identity to local system or network service, that account shows up in job 'Run As User' in cluster manager -> job management. All I want is the end user's login ID there. since he/she is part of HPC user list that can submit jobs.

    The advantage with this approach is obvious - I do not have to ship the HPC dll's and/or install HPC client utilities with my custom written application!!!

    Saturday, August 13, 2016 11:28 PM
  • Hi SRIRAM R,

    HPC Pack SOA via IIS has never been tested or officially supported, so we don't have any samples available around it. Anyway, this is an interesting scenario which we may consider to support in the future release.

    Meanwhile, as I checked the session API code, the current SOA client allowed impersonation level is set to Impersonation instead of Delegation. So if you would like to try it out now, I would recommend to place all the services (IIS, head node and broker node) and as well the service registration folder on the same headnode machine.

    If you would like to try private bits with alternative impersonation level, please send us email @ hpcpack@microsoft.com


    Yutong Sun

    Monday, August 15, 2016 2:16 PM
  • Thank Yutong. I have sent an email requesting the private bit.

    I will try out copying the my SOA dlls to headnode. As i said in my earlier post,  'my' WCF is in IIS which is on the headnode itself; so is the HPC SOA service registration file.. what is not on that headnode is the actual SOA dll..

    HPC SOA aside, What is weird is that I cannot create a IScheduler object using Impersonation from my custom WCF service inside of IIS that is installed on HeadNode ... Would you know why that may be happening?

    Just so I understand this correctly, HPC SOA framework *does not* support Delegation as it is today?  {I am okay with Impersonation, if that works fine,given that all of my services(IIS WCF/HPC Headnode/SOA registration file etc) are on the same box.}

    • Edited by SRIRAM R Monday, August 15, 2016 5:35 PM
    Monday, August 15, 2016 4:52 PM
  • I've sent you the private bits, Sriram.

    SOA service dlls are consumed by service hosts on compute nodes, which I suppose is not related to the session creation failure.

    When you say you cannot create a IScheduler object, is there any error happened?

    Yes, you are right, HPC Pack SOA does not support Delegation for current official releases (version <= 4.5.5111).


    Yutong Sun

    Tuesday, August 16, 2016 8:27 AM