none
3 HPC head nodes in High Availability with builtin Service Fabris RRS feed

  • Question

  • Hi , I have the HPC Cluster (2016 Update 2) with 3 head nodes configured in High Availability (Service Fabric) , one of the head nodes server dead and I had to re-install the OS from the begin  (Windows Server 2012).
    To be more precise I'm able to join back the head node to the cluster via the HPC Pack 2016 setup but the Service Fabric can't be installed over it so it appears Unknown  and Invalid to the Service Fabric Explorer and +"error" to the HPC Manager console.
    What is the right procedure to add,remove,repair an head node to the HPC 2016 configured in high availability with the builtin Service Fabric?

    Thank you.
    • Edited by Frizza Thursday, January 24, 2019 7:33 AM
    Wednesday, January 23, 2019 11:42 AM

All replies

  • Hi Frizza,

    Please follow the steps below to replace a bad head node,

    0. Prerequisites

    0.1 Make sure the new head node is joined the same domain as the old one, and has the same name

    0.2 Make sure the new head node has installed the required certificates

    0.3 Find ServiceFabric\MicrosoftAzureServiceFabric.cab in the HPC Pack setup, unzip it to obtain MicrosoftAzureServiceFabric\bin\Fabric\Fabric.Code\InstallFabric.ps1, copy the .ps1 and the .cab on the new head node

    0.4 Copy C:\ProgramData\SF\clusterManifest.xml from the old head node to the new one

    On the new head node, perform the following operations

    1. Install head node prerequisites

    2. Start Powershell as admin, run .\InstallFabric.ps1 -FabricRuntimePackagePath "MicrosoftAzureServiceFabric.cab"

    3. Restart the machine

    4. Start Powershell as admin, run New-ServiceFabricNodeConfiguration -ClusterManifestPath clusterManifest.xml

    5. Start FabricHostSvc service

    Regards,

    Yutong Sun


    Monday, January 28, 2019 3:07 AM
    Moderator
  • Thank you Yutong for the answer, another question , is the above procedure valid if I have to replace a head node with a new one with a different name ?
    Thank you.
    • Edited by Frizza Monday, January 28, 2019 5:20 PM
    Monday, January 28, 2019 7:55 AM
  • The given procedure above requires the new head node has the same name as the old one. It doesn't apply to adding a new head node with different name.

    Regards,

    Yutong Sun

    Tuesday, January 29, 2019 7:47 AM
    Moderator
  • Thank you again Yutong, in our company we are going to deploy HPC 2016 with High Availability head nodes but we want to be sure that there is a procedure in case  we have to replace a head node with a new one with different name.
    The config that we are going to deploy is:
     - HPC 2016 Update 2 in High Availability (service Fabric) with 3 head nodes per DC
     - Remote SQL Server DB in Fileover
     - 8 Compute nodes per DC

    I'm not able to find any documentation about how to replace an head node with a new one  with different name, could you help me with that?

     Thank you in advance.
    Tuesday, January 29, 2019 8:45 AM
  • Replacing a head node with a new one with different name has not been validated, thus currently not supported. You need to replace the bad head node with a new one with the same name in the same domain.

    Regards,

    Yutong Sun

    Tuesday, January 29, 2019 9:37 AM
    Moderator
  • Hi Frizza , we are trying to deploy HPC across datacetenters

    Our first attempt was not successful as we tried to have 3 headnodes across DC .. which was not correct as you need minimum 2 headnodes to be up 

    2 questions for you 

    in you case do 3 headnodes in each DC 3+3 =6 6 connect to same Remote DB and part of the same cluster or you have separate SQL Server in each DC ?

    Appreciate help

    thanks

    Julia


    • Edited by juliakir Sunday, January 19, 2020 4:17 PM
    Sunday, January 19, 2020 4:16 PM
  • Hi juliakir,

    for our specific needs we have created different dbs for each Headnodes cluster (DC), dbs were previously created ion a SQL Server cluster (always on availability groups) with different names for exemple HPCScheduler1 (DC1) and HPCScheduler2 (DC2)...

    Monday, January 20, 2020 7:42 AM
  • Hi Frizza, 

    thanks a lot for reply . Did you manually modify script created  by micorosoft to create  dbs with new names?

    appreciate your help

    thanks

    Julia


    • Edited by juliakir Monday, January 20, 2020 12:39 PM
    Monday, January 20, 2020 12:38 PM