none
SP2 beta sample for SOA Common-data API RRS feed

  • Question

  • I'd like to look at the new feature in the SP2 beta for "Common-data APIs for SOA workloads", and the release notes say there is a sample of it on Microsoft Connect. I downloaded the samples zip file from Connect but I don't see anything offhand. Where exactly is this sample?

    Friday, May 13, 2011 12:58 PM

All replies

  • The sample will be in the next release of SDK sample code. However, it missed the dead line of SP2 Beta.

    Meanwhile, let me at least post the code snippet on forum. You should be able to compile them using a Windows Console project to hold the client, and a WCF Service Library project to hold the service. Also remember to add reference to the SP2 beta SDK.


    Saturday, May 14, 2011 3:37 AM
  • // THIS CODE AND INFORMATION IS PROVIDED "AS IS" WITHOUT WARRANTY OF
    // ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO
    // THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A
    // PARTICULAR PURPOSE.
    //
    // Copyright (c) Microsoft Corporation. All rights reserved.
    
    using System;
    using System.IO;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using System.Reflection;
    // This namespace is defined in the HPC Server 2008 R2 SDK
    using Microsoft.Hpc.Scheduler.Session.Data;
    // This namespace is defined in the HPC Server 2008 R2 SDK
    using Microsoft.Hpc.Scheduler.Session;
    // This namespace is defined in the "CommonDataService" service reference
    using CommonDataClient.CommonDataService;
    
    namespace CommonDataClient
    {
      class Program
      {
        static void Main(string[] args)
        {
          string hostname = "shhpc03cn04";
    
          // DataClient Id used to identify the data, this should be unique across the cluster
          string raw_data_id = "raw_data_id";
          string dictionary_data_id = "dictionary_data_id";
    
          Console.WriteLine("Start. " + DateTime.Now.ToLongTimeString());
    
          // Create a DataClient to store a Dictionary data
          using (DataClient client = DataClient.Create(hostname, dictionary_data_id))
          {
            Console.WriteLine("Data {0} Created. {1} ", dictionary_data_id, DateTime.Now.ToLongTimeString());
            // Here we have a DataClient whose life cycle is not managed by SOA
            Dictionary<string, string> objects = new Dictionary<string, string>();
            objects.Add("key1", "value1");
            objects.Add("key2", "value2");
            // WriteAll() can only be called once on a data client. 
            client.WriteAll<Dictionary<string, string>>(objects);
          }
    
          SessionStartInfo sessionStartInfo = new SessionStartInfo(hostname, "CommonDataService");
          // Pass DataClient Id in SOA session's environment variable so that it could be read from service code
          sessionStartInfo.Environments.Add("DICTIONARY_DATA_ID", dictionary_data_id);
    
          using (Session session = Session.CreateSession(sessionStartInfo))
          {
            Console.WriteLine("Session {0} Created. {1} ", session.Id, DateTime.Now.ToLongTimeString());
    
            // Create a DataClient to store the raw data read from the file
            using (DataClient client = DataClient.Create(hostname, raw_data_id))
            {
              Console.WriteLine("Data {0} Created {1}. ", client.Id, DateTime.Now.ToLongTimeString());
              // Add a data life cycle management so that it'll be deleted when session is done
              // Otherwise, the data will have to be cleaned up by client
              client.SetDataLifeCycle(new DataLifeCycle(session.Id));
              // WriteRawBytesAll() doesn't serialize the object and will write the byte stream directly.
              // Use this when you want to transport a file or have non-.Net code that cannot handle .net serialization. 
              client.WriteRawBytesAll(File.ReadAllBytes("DataFile.txt"));
            }
    
            // Send/Receive SOA requests
            using (BrokerClient<ICommonDataService> client = new BrokerClient<ICommonDataService>(session))
            {
              Console.WriteLine("Send Requests. " + DateTime.Now.ToLongTimeString());
              client.SendRequest<GetDataRequest>(new GetDataRequest(raw_data_id));
              client.EndRequests();
              Console.WriteLine("Get Response. " + DateTime.Now.ToLongTimeString());
              foreach (BrokerResponse<GetDataResponse> resp in client.GetResponses<GetDataResponse>())
              {
                string result = resp.Result.GetDataResult;
                Console.WriteLine(result);
              }
            }
            Console.WriteLine("Start closing session. " + DateTime.Now.ToLongTimeString());
          }
          Console.WriteLine("Session Closed. " + DateTime.Now.ToLongTimeString());
          
          Console.WriteLine("Start cleaning up the Data {0}. {1} ", dictionary_data_id, DateTime.Now.ToLongTimeString());
          // We should delete the DataClient "dictionary_data_id" here since it's not managed by SOA
          DataClient.Delete(hostname, dictionary_data_id);
          
          Console.WriteLine("Press any key to exit.");
          Console.ReadKey();
        }
      }
    }
    
    
    Saturday, May 14, 2011 3:39 AM
  • using System;
    using System.Collections.Generic;
    using System.Linq;
    // THIS CODE AND INFORMATION IS PROVIDED "AS IS" WITHOUT WARRANTY OF
    // ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO
    // THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A
    // PARTICULAR PURPOSE.
    //
    // Copyright (c) Microsoft Corporation. All rights reserved.
    using System.Runtime.Serialization;
    using System.ServiceModel;
    using System.Text;
    using Microsoft.Hpc.Scheduler.Session.Data;
    using Microsoft.Hpc.Scheduler.Session;
    
    namespace CommonDataService
    {
      // NOTE: If you change the class name "CommonDataService" here, you must also update the reference to "CommonDataService" in App.config.
      public class CommonDataService : ICommonDataService
      {
        private static Dictionary<string, string> objects;
        static CommonDataService()
        {
          string dictionary_data_id = Environment.GetEnvironmentVariable("DICTIONARY_DATA_ID");
          // Use ServiceContext to obtain DataClient
          using (DataClient client = ServiceContext.GetDataClient(dictionary_data_id))
          {
            // Using static constructor and objects to share the data across all GetData calls saves memory and reading/deserialization cost.
            objects = client.ReadAll<Dictionary<string, string>>();
          }
        }
    
        /// <summary>
        /// To demo read common data in SOA service code
        /// </summary>
        /// <param name="raw_data_id">DataClient Id for the raw data</param>
        /// <returns>The result string shows the data information</returns>
        public string GetData(string raw_data_id)
        {
          string result;
    
          // One can also read and initialize the data in each service request.
          // Notice the IDisposable pattern here. Also dispose dataclient after using it.
          using (DataClient client = ServiceContext.GetDataClient(raw_data_id))
          {
            byte[] bytes = client.ReadRawBytesAll();
            result = bytes.Length.ToString() + "\n";
          }
    
          // Process data obtained in service static constructor
          foreach (KeyValuePair<string, string> obj in objects)
          {
            result += string.Format("{0}:{1}\n", obj.Key, obj.Value);
          }
          return result;
        }
    
      }
    }
    
    
    Saturday, May 14, 2011 3:40 AM
  • Thanks. I think I followed that, but when I try to run I get a DataException saying that "No data server is configured". Is there some prerequisite I need to install or something I need to configure?
    Monday, May 16, 2011 8:29 PM
  • Yes, you need to first configure common data file share by:

    cluscfg setenvs HPC_SOADATASERVER=<shareFolderPath>

    then restart the HpcSession service on headnode

    Make sure the user and headnode machine account (or SYSTEM account if the share folder is on headnode) has read/write access to the folder.

    Wednesday, May 18, 2011 1:26 AM
  • Thanks. It looks good when run in the on-premise nodes, but it fails for me when run on an Azure compute node. Is it expected to work on Azure nodes? I was hoping that was the point of the feature.
    Wednesday, May 18, 2011 8:07 PM
  • It's not workin on Azure for this release. And we are evaluating the possibility to extend it to Azure. We are still under planning phase for next version so nothing is certain now. (As you can see we are still working on SP2 now).

     

    What kind of data are you putting onto common data? And what kind of workload do you have for Azure?

     

    P.S. Those configuration hassles will go away in SP2 final release, common data will work out-of-box at that time.

     

    -yiding

    Thursday, May 19, 2011 2:36 AM
  • I'm trying to upload the user data files from the client app to the HPC node at runtime. The files can be from 4MB up to very large sizes. We want to have the cluster do many computations of the data file, using a GA service. So it's very compute heavy. I've recently rigged it to upload the files to an on-premise file share and/or Azure storage depending on the compute node types in the cluster, and then the compute nodes would read the files from their respective locations. It's not bad, but I was hoping for an abstract way of handing the data to the cluster without me handling on-premise/Azure logic manually.
    Friday, May 20, 2011 12:46 PM
  • Thanks for the feedback. Yes, this is the kind of scenario we want to support, in the future. Meanwhile, you can use the hpcpack/hpcsync utility to accomplish staging (to Azure).

     

    When you say it's 4MB to very large, how large it could be? Do you now use Azure Storage as a staging area?

    Monday, May 23, 2011 10:40 AM
  • Yeah, I'm using Azure Storage APIs to stage the data because the hpcpack stuff was very slow, and then you have to do it serially for each compute node. So that really wasn't an option.

     

    We have seen data files that are at the 2GB mdb limit, so while that is extreme it is something we need to handle.

     

    Monday, May 23, 2011 1:39 PM