none
Cluster storage requirements

    Question

  • When designing a cluster, what is the storage requirements with regards to analysis data? All the information I see on storage requirements seem to refer to what is required for the OS and HPC to run. I see mention of a file share for analysis data but nothing indicating whether this storage should be local on the cluster or can be on the Enterprise network. Is efficiency increased with local storage on the high speed network?

    Thanks,

    Wednesday, June 23, 2010 10:22 PM

Answers

  • Ah yes such a seemingly simple question :)

    This is a big subject, and the answer is very much dependent on how big your cluster is, and how you use it. Using an application which creates huge output files? Maybe running many Monte Carlo type simulations? Streaming massive input files? Running small MPI jobs which are not reliant on data read/write?

    You'll probably come to the conclusion that storage local to the cluster is more appropriate (lots of potential bottlenecks / contention between a cluster node and enterprise storage).

    I think it's worth starting with the following document, but it's very much a case of horses for courses. It may be that a simple share on your headnode will meet your requirements.

    http://resourcekit.windowshpc.net/AT%20A%20GLANCE/Papers1/WindowsHPCServerStorageOptionsandSolutions.pdf

    Cheers

    Dan

    Thursday, June 24, 2010 10:14 AM