Sign in
Microsoft.com
 
Microsoft
 
 
 

locked
Question regarding file I/O in Azure burst cluster RRS feed

 > 

  • Question

  • Question
    Sign in to vote
    0
    Sign in to vote

    I need to create an application that takes input files (each file a separate request), modifies them somewhat via WCF service and outputs the data.  I have already implemented a test of this on a small on-premise cluster.  Reading up on Azure, I can understand the details for a cluster completely on Azure.  However, I am stymied with regards to a mixed cluster.

    Ideally, I would not want to have two copies of the data as this can get unwieldy, and I would need the output in one place.  I would prefer to have the files in one, or the other location and for each job, copy the files over on an as-needed basis based on node type (AzureNode vs. Other).  Does anyone know of any reference I could use for this task? Is there a way I can programmatically tell which node-type I'm running on?  In addition to transferring input, I would need to also move the subsequent output. 

    Thursday, March 12, 2015 8:15 PM

Answers

All replies

  • Question
    Sign in to vote
    0
    Sign in to vote

    Hi KWilliams,

      We actually supports hybrid cluster (HPC Burst to azure) as well as deploying the whole cluster in azure. So please check more details on the HPC burst.

      With HPC Burst, we provide ways for data movement between on-premise and cloud through hpcpack and hpcsync. . With HPC burst, you will know where your task is running. If you use SOA, the common data scenario will move the data to cloud transparently for your SOA job... Please let us know if you have further questions on HPC burst solution

      As for your ideal expectation, this is one of the area we are currently working on in the upcoming release. If you're interested in providing more feedbacks and willing to try out our invite only preview, please contact us through hpcpack@Microsoft.com

    Qiufang Shi

    Monday, March 16, 2015 3:06 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Thank you for the reply.  My first question would be, is there an API for getting the node type.  For example, if I were to have a service that would take in all the necessary input and then be able to handle both cases on the fly.  For example

    If (this.node = AzureWorkstation)

      return AzureSOACall(inputvariables);

    else

      return LocalSOACall(inputvariables);

    I would like to remain maximally flexible.  Therefore the hypothetical user could have their information stored locally or stored in the cloud and the code would handle things accordingly.  Otherwise, the code will have to dictate where the input file stores would be, which would be less feasible, helpful for users.

    Thank you for the information, I will look into the preview.


    • Edited by KWilliams1 Monday, March 16, 2015 3:28 PM
    Monday, March 16, 2015 3:25 PM
  • Question
    Sign in to vote
    0
    Sign in to vote

    1. To know whether you're running on azure, please try to check below Environment whether exists and value is 1:

    CCP_ONAZURE=1

    2. As you are using SOA, you can also check the common data feature: http://blogs.technet.com/b/windowshpc/archive/2013/03/14/hpc-pack-soa-tutorial-iv-common-data.aspx For it to be work for azure nodes, you need configure a storage connection string: http://go.microsoft.com/fwlink/p/?LinkId=267093

    Qiufang Shi


    Tuesday, March 17, 2015 2:15 AM
  • Question
    Sign in to vote
    0
    Sign in to vote

    Thanks for the information.  I already have an Azure storage account and have added the SOA DLL and input files to my Azure node template.  This way they are uploaded automatically every time I provision an Azure workstation.  For my current tests, I can simply upload all input files to the Azure account and use the CCP_PACKAGE_ROOT environment variable to find them. Adding CCP_ONAZURE, I can find the input file on all workstation (if envvar = 1 packageroot\file else \\servershare\file)

    However, I still want to be able to have a single store of input files and move them back and forth between cloud and local on an as needed basis. 

    Wednesday, March 18, 2015 2:42 PM
  • Question
    Sign in to vote
    0
    Sign in to vote
    The requirement you're asking will be available in our next release but it is now available in current official release. So please stay tuned. The next release will be a few months away.

    Qiufang Shi

    Friday, March 27, 2015 6:57 AM
  • Question
    Sign in to vote
    0
    Sign in to vote
    I'm not sure I understand.  You state that it will be available and is currently available at the same time.  Could you please clarify.
    Friday, March 27, 2015 12:09 PM