DurableSession.CreateSession apparently hangs with new cluster [SOLVED] RRS feed

  • Question

  • I know this is probably just the way Windows Authentication behaves, but I just spent two hours going crazy over this.

    I have an HPC service that's working fine in my own test lab.

    I now have a new cluster that's just been set up for testing, and I've deployed my service to it. I run my test client and... nothing. The tests simply hang forever, it seems.

    When I debug, it's clear that the test is hanging in DurableSession.CreateSession, with no apparent reason.

    I switch my tests back to using the old cluster, and they work fine.

    Eventually I go back to square one and run the EchoService sample test against the new cluster - same hang, but since this is a Console application, I notice the prompt "Enter password for [DOMAIN]\[user]:". After entering my password and answering "Y" for "Save password?", the EchoService responds. My tests also now work fine against the new cluster.

    Obviously, authentication is required to access the new broker, but I'm concerned that when I roll out HPC support to my customers I'm going to get many calls out of the gate that the service doesn't respond. Shouldn't there be some kind of UI prompt when not running as a console app? In this case if I hadn't decided to go back to the EchoService I'd probably still be debugging this.

    What can I do to prevent this for new clusters?


    Thursday, August 25, 2011 3:15 PM

All replies

  • Argh. I just got burned by this again after my domain password reset. Does anyone have an idea about how to avoid this in production scenarios?


    Tuesday, September 6, 2011 3:37 PM
  • I found another thread from a guy with the same problem: http://social.microsoft.com/Forums/en-US/windowshpcdevs/thread/d8a39951-a7f4-4192-8f76-5a11e0fa7dc3

    That thread was closed out without a resolution.

    Tuesday, September 6, 2011 7:26 PM
  • OK, I think I'm getting closer, but still not understanding the design rationale here. Now I've become aware of "cluscfg setcreds", I logged into the head node and ran this command in a HPC powershell, providing the headnode, my domain username, and password.

    Now, when I run the CLIENT (console) I don't get prompted for the password.

    I'm still concerned about how this is intended to work in a corporate production environment, with the session/broker client "embedded" in a desktop application. If I have not previously run "cluscfg setcreds" on the head node for the user who runs the desktop application, that application will appear to hang while credentials are being silently requested.

    I don't know, I was kinda expecting that if the domain user running the app (invoking the service) is in the AD group of users that are "Cluster users", that the authentication challenge would be unneccessary.

    Am I supposed to have every user login to the headnode and run "cluscfg setcreds" before they start using my application (and thereafter every time AD policy insists that they change their password)?

    What am I missing?

    • Edited by wbradney Tuesday, September 6, 2011 7:59 PM
    Tuesday, September 6, 2011 7:58 PM
  • Hi wbradney,

    Take a look at the Session.SetInterfaceMode method (http://msdn.microsoft.com/en-us/library/microsoft.hpc.scheduler.session.session.setinterfacemode%28VS.85%29.aspx).

    We use it like this

        Session.SetInterfaceMode(false, (System.IntPtr)0);

    to open a dialog window in a GUI application for a user to enter their credentials in case they are not already cached.


    Wednesday, September 14, 2011 3:17 PM
  • Hi wbradney,

    Take a look at the Session.SetInterfaceMode method (http://msdn.microsoft.com/en-us/library/microsoft.hpc.scheduler.session.session.setinterfacemode%28VS.85%29.aspx).

    We use it like this

        Session.SetInterfaceMode(false, (System.IntPtr)0);

    to open a dialog window in a GUI application for a user to enter their credentials in case they are not already cached.


    Thanks, Derek - I'll check that out.

    I'm still not sure that my customers are going to be pleased by being asked to provide the same domain credentials twice, though, even if I can find a clean way to embed it into the workflow. Also, I wonder what happens with SetInterfaceMode when the credentials have already been cached, but have expired due to a domain password policy?

    • Edited by wbradney Thursday, May 24, 2012 9:18 AM
    Thursday, September 15, 2011 12:20 AM
  • If the password for the cached credentials has expired, then the next time that user tries to submit to HPC, the dialog will open allowing the user to enter the new password.

    The dialog does allow it to remember the credentials, so it is not necessary for this dialog to appear every time.

    Thursday, September 15, 2011 4:00 PM
  • I finally got around to trying out Session.SetInterfaceMode and it works as described by Derek.

    I'd still like to understand why a separate credential check is necessary for submitting HPC jobs, if I'm already logged in to Windows with the same domain credentials...

    Also, this arrangement has the unfortunate consequence of requiring high-level application code to take a dependency on the HPC API in order to declare that they're either a console or UI application, even if they never actually get around to running a HPC job (in my case grid computing services are abstracted behind a provider model that offers several different grid platforms and can be used in console, UI or scripted apps). I'll probably punt on this and always assume that the HPC client is a UI...
    • Edited by wbradney Wednesday, May 30, 2012 2:21 PM
    Wednesday, May 30, 2012 1:36 PM
  • I still see this problem. It's not resolved as of HPC 2012 R2 update 3 (with june 2016 fixes)

    Wish someone at MS was listening to feedback from customers. 4 years and the problem still persists - phew!

    (I just had to reset my password yesterday due to our AD password policy and today i see this problem - job is submitted under the username, but its stuck in configuring). once i set the setinterface mode and get the latest password from the user, two jobs are created instead of one! go figure!

    • Edited by SRIRAM R Wednesday, July 13, 2016 2:18 AM
    Wednesday, July 13, 2016 2:07 AM
  • Hi SRIRAM R,

    Sorry, this is currently by design that the user need to reenter his/her password if the cached credential has been expired, either from command line or GUI. Users may use Set-HpcSoaCredential powershell command to save the client credential before creating new sessions after the password is changed, however once an expired or incorrect credential is provided, the prompt in command line or GUI should be there for the new or correct password.

    As for the redundant Configuring job created when using a new credential, this is a known issue caused by the session creation logic. The session client would first try to submit the Configuring job but would fail because of the expired credential, after the new credential is provided, the session client would not reuse the Configuring job created previously, but just create a new Configuring job and then submit it. We did not fix this issue because the impact seems small.


    Yutong Sun

    Wednesday, July 13, 2016 1:13 PM
  • I have a C# based windows app that creates DurableSession() object.

    Two issues:

    1) With SetInterfaceMode(), the second job that is created marks all jobs as 'failed' and sets the job to faulted state (i mention this in another thread); and first job sits in 'configuring' state

    -- how long does the first job sit in configuring state before it is marked as completed or cancelled (by the job scheduler automatically?)

    2) Explicitly setting username/password in SessionInfo object - Same issue as above - One job is created (as expected), but very soon all tasks are failed and job goes to faulted state.

    edit: I put a trace in the very first line of my SOA dll's method and that is not executed - something to do with the dll itself not being loaded perhaps? (i have a preparenodecommand which run an exe from the saem folder as SOA dll and that seems to be working fine..) 

    strange things are happening since i changed my AD password.

    • Edited by SRIRAM R Wednesday, July 13, 2016 2:56 PM
    Wednesday, July 13, 2016 1:28 PM
  • 1) For the first job in 'Configuring' state, it won't change any more if user don't make any further operation. The scheduler won't schedule any resource for 'Configuring' jobs before they are submitted and enter the 'Running' state.

    2) It is also possible that broker failed to send the request to the service host. If you doubt whether the service code is loaded, you may write log in the service constructor and see if it is executed.


    Yutong Sun

    Thursday, July 14, 2016 1:46 AM
  • So do I need to call DurableSession.SetInterfaceMode(false,(IntPtr)0) everytime before DurableSession.CreateSession or does DurableSession.CreateSession automatically invoke/show the credentials dialog to the user whenever the 'cached' credentials expire? (Mine is a Windows Forms application)
    • Edited by SRIRAM R Friday, July 15, 2016 4:05 AM
    Friday, July 15, 2016 4:04 AM
  • SetInterfaceMode is a static method that needs to call only once in the client application process for the following CreateSession calls.


    Yutong Sun

    Friday, July 15, 2016 7:21 AM
  • I get that, but my question was can I avoid using the SetIntefaceMode altogether? Is DurableSession.CreateSession() smart enough to display the credentials dialog when the cached credentials expire?

    What I meant by 'everytime' is if the user is submitting different jobs based on different inputs, the credentials dialog will popup before CreateSession() is called, which is redundant in my opinion

    Friday, July 15, 2016 12:50 PM