none
Right way to make disks available to jobs running on other node

    Question

  • What is the right way to make raid disks attached to the head node permanently available for all users or groups of users where their jobs will typically be spread across all nodes? Prefer method that also gives fastest transfer.

    Scenario:

    HPC cluster with 4 nodes, running HPC Server  (pre R2 version)

    3 teams of users who run jobs on the HPC, 4 Raid drives  nominally assigned 1 per team of users

    Typically each instance will copy some data from the teams raid drive to the nodes D drive, process for 5-30 minutes, and copy results back to raid drive. In some cases a job consists of 1000 instances ( i.e. sweep from 1..1000)  We are only getting ~50% CPU utilization on the non-head-nodes, this appears to be due to network latency accessing the raid drives.

    We have been using net use command to map the raid drive to a drive letter, but this doesn't seem to reconnect and I would prefer a method that works better.

    I'm new to this kind of Windows admin, and to Win HPC, can anyone tell me the way we are supposed to do this ?  Thanks

     

     

    Friday, December 17, 2010 12:17 AM

All replies

  • Hi Kbam, if you want the mapping drive exists during the task execution on that node, you can add Node preparation/release tasks, here is an example job:

    - Add PreTask: net use w: \\UNCPath

    - Add you working task, where you can access w:

    - Add RelTask: net use w: /delete /Y


    Qiufang Shi
    Thursday, December 23, 2010 3:38 AM
  • Thank you Qiufang Shi for the reply,

    Unfortunantly  net use   doesn't seem to work the way that I want. "We have been using net use command to map the raid drive to a drive letter, but this doesn't seem to reconnect "

    I want to "make raid disks attached to the head node permanently available for all users"

    i.e. when ever a job starts the disk will be automatically mounted ( if it isn't already) a bit like mapping a network drive on my laptop, but how do I do that for users who will only run jobs and dont have command line access. and will it work ? work well ? or is there some better way to do achieve what I want ?

    If net use IS the best way then I guess that is what I will have to do, but if there is a better way then I would love to hear it.

    Thanks,

    Kbam

    Thursday, December 23, 2010 7:43 AM
  • I might simply create a share on the RAID array and have the job use a UNC path...  e.g. '\\headnode\RaidShare'.

    Does the job require that the share be mapped to a drive letter?

    --Brian

    Sunday, January 02, 2011 8:40 AM