none
Scheduler not executing Excel Cluster Pack Workbook Offloading RRS feed

  • Question

  • I have followed https://docs.microsoft.com/en-us/azure/virtual-machines/windows/excel-cluster-hpcpack and created a cluster using 'HPC Pack cluster for Excel workloads'. The desktop calculation works but I encounter 2 issues when running the cluster option which deals with ExcelClient.Initialize(ExcelWorkbook as unknown [,dependFiles as string]), specifically the dependFiles parameter.

    When dependFiles is passed 'ExcelClient.OpenSession()' throws 
     'Unknown error occured: System.UnauthorizedAccessException: Access to the path '\\\\<headNodeName>\\Runtime$\\SOA\\UserJobs\\<username>\\<username>_<32DigitHexKey?>' is denied.
       at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
       at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
       at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
       at System.IO.File.InternalWriteAllBytes(String path, Byte[] bytes, Boolean checkHost)
       at Microsoft.Hpc.Scheduler.Session.Internal.SessionLauncher.SessionLauncher.AllocateInternal(SessionStartInfoContract startInfo, String endpointPrefix, Boolean durable, Int32& sessionid, String& serviceVersion, SessionInfoContract& sessionInfo).'. 

    When dependFiles is NOT passed the Joy Manager acknowledges and receives the job. I can see the state go through: Configuring, Validating, 'State = Queued' and 'Pending Reason = Not enough available resources.' and finally Running. It never gets passed 'Running' and within the module 'CalculateWorkbook' it proceeds to the line 'Exit Sub' to finish. 'HPC_GetVersion' is the only HPC method called. None of the following are called 'HPC_Initialize', 'HPC_Partition', 'HPC_Execute', 'HPC_Merge', 'HPC_Finalize'.

    I'm including this in case it helps to diagnose the issue. I'd previously been getting the following message before I added a 'Generic Credential' through 'Control Panel' :   'BrokerPersistQueue flush timeout for waiting for the incoming requests within 60000 milliseconds for client[clientID=<37CharacterString>].

    System.TimeoutException: BrokerPersistQueue flush timeout for waiting for the incoming requests within 60000 milliseconds for client[clientID=<37CharacterString>].

    at Microsoft.Hpc.Scheduler.Session.BrokerClient`1.Flush(Int32 timeoutMilliseconds, Boolean endOfMessage)

    at Microsoft.Hpc.Excel.ExcelClient.CalculateCluster()

    at Microsoft.Hpc.Excel.ExcelClient.Run(Boolean executeLocally)'

    'HPC_Execution' was never called, nor was 'HPC_Merge'. However, 'HPC_Finalize' was called and did finish to completion.

    I'm out of my element hear, bear with me if I didn't explain well enough.

    Tuesday, June 20, 2017 9:33 PM

Answers

  • Hi ExcelGrunt,

    This access unauthorized error was caused by using <username> instead of <domainname>\<username> for the param UserName in the HPCExcelClient.OpenSession call in Excel client workbook. So the access right to write depending files under the UserJobs\<username> folder for the domain user was not granted. If specifying '<domainname>\<username>' for the UserName, the folder would be created as UserJobs\<domainname>.<username> and the domain user would have the correct read/write access.

    Regards,

    Yutong Sun

    Thursday, June 22, 2017 2:36 PM

All replies

  • Hi ExcelGrunt,

    Is the HPC cluster created on Azure with the ARM template or the deployment script? The dependFiles is used to move the Excel workbooks (and other depending files) from the on-prem client machine to the cluster head node on Azure. The HpcSession service running on the head node should have the full access right to the runtime data share folder ('\\\\<headNodeName>\\Runtime$\\SOA\\') once the Azure cluster is successfully deployed, and should be able to create sub folders with correct access rights for the job owner. Please check the folder '\\\\<headNodeName>\\Runtime$\\SOA\\UserJobs\\<username>\\<username>_<32DigitHexKey?>' on the head node to see if this folder was created with correct security settings, especially full control for Local System account.

    When running the Excel workbook offloading on a cluster, 'HPC_Execute' macro shall be called in the Excel workbook on the compute nodes to perform the calculation. 'HPC_Merge' would be called in the Excel workbook on the client once the calculation results are returned from the compute nodes. It looks there is no responses returned so the 'HPC_Merge' was not called.

    Regards,

    Yutong Sun

    Wednesday, June 21, 2017 9:55 AM
  • Yutong,

    The cluster on azure was not created with a deployment script. It was set up using the deployment model 'Resource Manager'. I left any default settings as they were set up, the only change I made was to set up with 3 compute nodes.

    I checked the folder '\\hpcproduction\Runtime$\SOA' and only have read-access to it. I cannot create sub-folders nor files within there. There was no folder '\\<headNodeName>\Runtime$\SOA\UserJobs\<username>\<username>_<32DigitHexKey?>'. There are 2 folders found within `UserJobs` '<username>' which had a single sub folder '33' and an erroneously typed <badUsername> that I incorrectly typed when trying to do the initial log on. How do I go about providing read-write access? I was able to create folders and files in '\\hpcproduction\Runtime$'.

    Thanks for explicitly clarifying that 'HPC_Execute' will be called on the compute nodes.


    • Edited by ExcelGrunt Wednesday, June 21, 2017 5:37 PM
    Wednesday, June 21, 2017 5:05 PM
  • Hi ExcelGrunt,

    This access unauthorized error was caused by using <username> instead of <domainname>\<username> for the param UserName in the HPCExcelClient.OpenSession call in Excel client workbook. So the access right to write depending files under the UserJobs\<username> folder for the domain user was not granted. If specifying '<domainname>\<username>' for the UserName, the folder would be created as UserJobs\<domainname>.<username> and the domain user would have the correct read/write access.

    Regards,

    Yutong Sun

    Thursday, June 22, 2017 2:36 PM
  • Yutong,

    That was a step in the right direction for me. The scheduler receives the request and can now process them. I'm refactoring my code to run correctly. If I have issues related I'll create a new post and link to it here. 

    Thank you for your help, it has been invaluable.

    Friday, June 23, 2017 4:00 PM