Can MPI take raw data stream? RRS feed

  • Question

  • This is also posted in the developer section as I wasn't sure which forum would be the best to find the solution to my issue....any help VERY appreciated, TIA!

    We have CCS 2003 SP1 implemented currently, but may need to move to HPC 2008 for this, which is part of the question.  The goal is to take a streams of raw data, currently transported by tcpip/winsock, convert this into something that can be passed through the/a scheduler so that it can manage near real-time analysis, very minimal latency through the cluster which may alter the stream slightly, and then be able to convert and send traffic back through to the raw data interface.

    There have been limitations with processing through one serial channel, so we'd like to run multiple channels and we think we can produce what we need, as long as we can get the data in and out fast enough. 

    Right now we have a small cluster, but would only be used for this one purpse, 2 nodes, quad-cores, with 8Gb of RAM each, Cisco 2960G, teamed NIC interfaces, and TOE cards as well.   All completely isolated to this task and the source/destination interface plugged into the switch.  Although I'm not sure if this is enough firepower to do it?

    We intend to run a job that opens a socket and streams the data into memory, which would be very small, but lots of data passing, still only 2Gb for the entire duration, each stream would represent one channel, which we hope can be spread across multiple nodes.  So this brings us to the challenges....

    How do we write the interface that recieves the data?  Does this need to run through MPI?  Can the interface be real-time, or does it have to be batched in?  i.e. do we need to try and break the data up in smaller transactions and batch it through to get the result?  How do we get the cluster to accept the data stream and allow us to view it and manipulate it?

    Thanks so much for your help on this!
    Thursday, April 9, 2009 3:47 AM


  • Hi - I am not sure I understand what you are doing. Is this processing a video stream? There are many products on the market that do video analytics. A quick search can turn them up.

    Wednesday, May 13, 2009 12:16 AM