none
How to code real-time stream? RRS feed

  • Question

  • We have CCS 2003 SP1 implemented currently, but may need to move to HPC 2008 for this, which is part of the question.  The goal is to take a streams of raw data, currently transported by tcpip/winsock, convert this into something that can be passed through the/a scheduler so that it can manage near real-time analysis, very minimal latency through the cluster which may alter the stream slightly, and then be able to convert and send traffic back through to the raw data interface.

    There have been limitations with processing through one serial channel, so we'd like to run multiple channels and we think we can produce what we need, as long as we can get the data in and out fast enough. 

    Setup:
    Right now we have a small cluster, but would only be used for this one purpse, 2 nodes, quad-cores, with 8Gb of RAM each, Cisco 2960G, teamed NIC interfaces, and TOE cards as well.   All completely isolated to this task and the source/destination interface plugged into the switch.  Although I'm not sure if this is enough firepower to do it?

    We intend to run a job that opens a socket and streams the data into memory, which would be very small, but lots of data passing, still only 2Gb for the entire duration, each stream would represent one channel, which we hope can be spread across multiple nodes.  So this brings us to the challenges....

    How do we write the interface that recieves the data?  Does this need to run through MPI?  Can the interface be real-time, or does it have to be batched in?  i.e. do we need to try and break the data up in smaller transactions and batch it through to get the result?  How do we get the cluster to accept the data stream and allow us to view it and manipulate it?

    Thanks so much for your help on this!
    Thursday, April 9, 2009 3:41 AM

Answers

  • Depends on your data pattern, the SOA platform delivered in HPC 2008 might be a good choice.

    With HPC SOA, you deploy you calculation code to all compute nodes on the cluster, setup a broker on head node, connect to the broker, transport calculation requests (and data) over WebService interface to the broker, have them processed on all compute nodes, and have the result sending back. HPC SOA use WCF to serialize and transport data so the latency will be less than 5 ms if you are a all-windows shop.

    To have an understanding on our HPC SOA platform quickly, here is a good video presentation

    http://channel9.msdn.com/shows/The+HPC+Show/Windows-HPC-WCF-Integration/ (total 4 parts)
    Thursday, April 9, 2009 6:02 AM

All replies

  • Depends on your data pattern, the SOA platform delivered in HPC 2008 might be a good choice.

    With HPC SOA, you deploy you calculation code to all compute nodes on the cluster, setup a broker on head node, connect to the broker, transport calculation requests (and data) over WebService interface to the broker, have them processed on all compute nodes, and have the result sending back. HPC SOA use WCF to serialize and transport data so the latency will be less than 5 ms if you are a all-windows shop.

    To have an understanding on our HPC SOA platform quickly, here is a good video presentation

    http://channel9.msdn.com/shows/The+HPC+Show/Windows-HPC-WCF-Integration/ (total 4 parts)
    Thursday, April 9, 2009 6:02 AM
  • Hi Jeff,

    What is your data source?

    You don’t have to use MPI to receive data, however if your processes need to collaborate to process the incoming data, MPI is a good parallel framework for that. If the processes are really independent, you can run many instances of your process on the cluster (using mpiexec or just the job-scheduler) in a long running job, where each process could listen on a socket port. However as you correctly said, your challenge is how to load balance across the servers. You have several options, from smart clients that know how to pick the server (randomly) to a load balancing network gear that picks the server base on the network traffic.

    Hope this helps,
    .Erez

    Monday, April 20, 2009 9:07 PM
  • Hi Lio,

    We are also facing a similar issue. We would like to send a video stream 24/7 to resource manager.

    Is it possible in Microsoft HPC?

    Thanks

    Anand
    Sunday, May 10, 2009 7:34 AM