Sign in
Microsoft.com
 
Microsoft
 
 
 

locked
Best Approach for HPC on very parallel workload RRS feed

 > 

  • Question

  • Question
    Sign in to vote
    0
    Sign in to vote

    I have a genetic algorithm application that I am currently using BizTalk to run for scatter gather processing and was trying to figure out if HPC might be a better fit. 

     

    Basically I need to:

    1) create a population

    2) run the algorithms

    3) asses the health

    4) cross breed

    5) repeat 2-4 many many many times 

     

     

    I am grappling with what exactly HPC will help me with .  Step 2 is the really big hit here and each algorithm can run in parallel (though they do use the same input dataset).  The new Parallel stuff in .NET 4 looks good, but I am not totally sure how this fits into HPC or if there is an equivalent analogy that would let me Parallel.For across an HPC cluster rather than just one server.  It seems like HPC SOA will require my client to be responsible for parallel calls and I almost wonder if the overhead of making service calls via WCF is worth it.  Currently I partition my problem and send out the work load batched and it sounds like HPC SOA will require me to do the same.  Am I correct here (that it will not partition for me)?  I think this is important because I am dealing in very large population sizes and they’re not that big (1-4k per algorithm) so making a service call to process one seems like it would result in a lot of thrashing. 

     

    I guess at the very least this can still make my life a lot easier as the world will be done on HPC nodes rather than BizTalk servers (which are not cheap or all that easy to manage).  And it’s a good excuse to play / benchmark some. 

    King Regards,

    -Dan

    Monday, January 10, 2011 8:27 AM

All replies

  • Question
    Sign in to vote
    0
    Sign in to vote

    A couple of issues will impact how to use HPC SOA (sepcifically, the message pattern):

    - Total number of request. How many individual in your population.

    - What's the message size. For each individual, how big is the data.

    - Calculation lenght. How long it will to calulcate fitness for each individual.

    - How big will your cluster be. How many cores/nodes will you have?

     

    Normally, HPC SOA will handle a couple of thousands requests per second. Beyond that, you might need to batch your calculation to avoid network overhead. Download HPC2008R2.SampleCode.zip from http://www.microsoft.com/downloads/en/details.aspx?FamilyID=BC671B22-F158-4A5F-828B-7A374B881172&displaylang=en and take a look at the SOA samples. The AsianOption project demonstrate the usage of HPC SOA to do Monte Carlo simulation, which will provide a good understanding on how to use the API.

     

    • Proposed as answer by Don Pattee Saturday, February 5, 2011 12:45 AM
    Monday, January 10, 2011 4:56 PM