none
HPC 2016.1 SDK (6088) can't connect to hpc 2012r2 node

    Question

  • Hello, I have a hpc 2012 R2 head node and I cannot connect from a hpc 2016.1 sdk ( version 6088)

    the following code 

                    try
                    {
                        scheduler.Connect(headnode);
                        Console.WriteLine($"server version: {scheduler.GetServerVersion().Major}.{scheduler.GetServerVersion().Build}");
                    }
                    catch (Exception ex)
                    {
                        Console.WriteLine(ex.Message);
                        throw;
                    }

    is catching an aggregate exception with inner exception : 

    An error occurred while sending the request. and 

    Return argument has an invalid type.

    Server version : 4.2.4400.0

    SDK version : 5.1.6088

    Any help would be appreciated, 

    Antoine

    Monday, 15 January 2018 12:57 PM

All replies

  • Hi Antonie,

    We can not repro this issue. Could you give us more information like:

    1. What's your client-side .net framework version?
    2. The detail information about the aggregate exception, include the inner exception type, call stack, etc

    Thanks,
    Zihao

    Tuesday, 16 January 2018 3:40 AM
  • Hello Zihao, 

    our .Net client side version is 4.7.1 (461310)

    the exceptions come as follow 

    Inner exception 1 : 

    + InnerException {"The underlying connection was closed: An unexpected error occurred on a send."} System.Exception {System.Net.WebException}

    Stack Trace : 

       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.HttpClientExtension.<>c__DisplayClass5_0.<<GetHttpApiCallAsync>b__0>d.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.RetryManager.<InvokeWithRetryAsync>d__33`1.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at Microsoft.Hpc.RetryManager.<InvokeWithRetryAsync>d__33`1.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.HttpClientExtension.<GetHttpApiCallAsync>d__5.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.HpcFabricRestContext.<>c__DisplayClass14_1`1.<<GetFirstHttpApiCallResponseAsync>b__0>d.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.HpcFabricRestContext.<GetFirstHttpApiCallResponseAsync>d__14`1.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.HpcFabricRestContext.<ResolveSingletonServicePrimaryAsync>d__10.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.ServiceResolverExtension.<ResolveSchedulerNodeAsync>d__11.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.Scheduler.Store.StoreConnectionContext.<ResolveSchedulerNodeAsync>d__55.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.Scheduler.Store.StoreServer.<ConnectWcfAsync>d__39.MoveNext()
    --- End of stack trace from previous location where exception was thrown ---
       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
       at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
       at Microsoft.Hpc.Scheduler.Store.StoreServer.<InternalConnectAsync>d__38.MoveNext()

    exception 2 : 

    - [1] {"Return argument has an invalid type."} System.Exception {System.InvalidCastException}

       at System.Runtime.Remoting.Proxies.RealProxy.ValidateReturnArg(Object arg, Type paramType)
       at System.Runtime.Remoting.Proxies.RealProxy.PropagateOutParameters(IMessage msg, Object[] outArgs, Object returnValue)
       at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
       at Microsoft.Hpc.Scheduler.Store.ISchedulerStoreInternal.Register(String clientSource, String userName, ConnectionRole role, Version clientVersion, ConnectionToken& token, UserPrivilege& privilege, Version& serverVersion, Dictionary`2& serverProps)
       at Microsoft.Hpc.Scheduler.Store.StoreServer.RegisterWithServer()
       at Microsoft.Hpc.Scheduler.Store.StoreServer.RegisterEvent(String schedulerNode)
       at Microsoft.Hpc.Scheduler.Store.StoreServer.ConnectWithRemoting(CancellationToken token)
       at Microsoft.Hpc.Scheduler.Store.StoreServer.<InternalConnectAsync>d__38.MoveNext()

    I tried on a new 2012r2 environment with a setup from scratch, and on our working 2012r2 environment the stacks and exceptions are the same

    Thank you very much, 

    Antoine

    Monday, 19 February 2018 11:17 AM
  • Hi Antoine,

    The first error "The underlying connection was closed: An unexpected error occurred on a send." is expected if you connect to a V4 cluster without specifying which connecting method to use. As SDK will first try to treat remote cluster as a V5 cluster and connect to it via WCF. If WebException happened, it will then try to connect again via remoting.

    This second error "Return argument has an invalid type" could be related with assembly resolution problems. Are you using HPC SDK via COM? If you enable fusion log on your client machine, is there any error it reports? Are Microsoft.Hpc.Scheduler.dll, Microsoft.Hpc.Scheduler.Session.dll and Microsoft.Hpc.Scheduler.Store.dll located by CLR the assemblies in the same folder with your client executor?

    Thanks,
    Zihao

    Thursday, 22 February 2018 4:50 AM
  • Hi Zihao,

    I have the same kind of issue. I used the HPC Pack SDK 5.1.6124 and client works for HPC 2016 Head node.

    But it can not connect with HPC 2012 Head node. The exception is: The underlying connection was closed: An unexpected error occurred on a send

    Any suggestions? Thanks.

    Tuesday, 9 October 2018 5:19 PM
  • Hi,

    Are you using HPC SDK via COM? Any error can be find in event log and fusion log?

    Thanks,
    Zihao

    Wednesday, 10 October 2018 1:48 AM
  • Thanks Zihao. 

    We have product (service and client) working with HPC 2008 R2 and HPC 2012. Now we try to work with HPC 2016 Update 1.

    The client code is not using COM.  It is c++ code calling .NET HPC API. I only saw this exception. I did not see other errors from fusion log.

    Currently the client side is binding with dlls from HPC 2016 SDK (5.1.6124). Service registered in HPC 2012 was built with old HPC SDK.


    Wednesday, 10 October 2018 8:50 PM
  • Hi,

    Thanks for the detailed information.

    If you are using C++ calling .net API, there is a great chance you're using COM. Could you show us the snippet?

    Also, could you try to copy assemblies in the SDK into the same folder of your exe file, including Microsoft.Hpc.Scheduler.dll, Microsoft.Hpc.Scheduler.Session.dll and Microsoft.Hpc.Scheduler.Store.dll and retry?

    Thanks,
    Zihao

    Thursday, 11 October 2018 1:27 AM
  • Thanks Zihao,

    We use C++/CLI to call .Net API. 

    I have assemblies including Microsoft.Hpc.Scheduler.dll, Microsoft.Hpc.Scheduler.Session.dll and Microsoft.Hpc.Scheduler.Store.dll with exe file. Version is 5.1.6124.

    So dll loading has no issue. But got exception: The underlying connection was closed: An unexpected error occurred on a send.

    Thursday, 11 October 2018 2:46 PM
  • Hi,

    Could you show us the exception detail, or a minimum code snippet to repro this?

    Thanks,
    Zihao

    Friday, 12 October 2018 1:19 AM