none
Python and UNC Paths RRS feed

  • Question



  • So I though i would be slick and write a python script to run my jobs across the cluster. Turns out when you submit a UNC path through os.system it does not work. os.system invokes CMD.exe which for some really annoying reason does not use UNC paths. Any thoughts on a work around

    Then, I said ok. I am smarter than the machine. So I tried to copy my executables all to C:\ on each node of the cluster but for some reason clusrun cannot exectute the simple COPY command when -recurse is given as option. Anyone know why?

    So finally put  mpiexec -n **  if fron of my executable myexecutable, even though my executable is not a parralel job.  It worked but it turns out my executable need a  preprocessor to preprocess the job (Cl /EP) and I don't have Cl.exe installed across the whole cluster. Only on the head node  and not in a shared directory. Is there an msi out there in which I can install Cl.exe across the whole cluster using clusrun? I have 32 nodes and do not want to spend 20 minutes per node installing visual studio on each node.

    Thanks,

    Ilya




    Saturday, February 2, 2008 8:57 PM

Answers

  • First off, cmd.exe definitely should support UNC paths.  Can you provide some sample of how you're trying to use it?  If you do something of the form:

    Code Snippet
    copy "\\server\share\some file.dat" "some other file.dat"

    That should work!

     

    As for the second questions, -recurse is not a valid flag accepted by copy.  I believe copy will just copy whole subtrees by default if you point it to a directory.  But for more granular control over what happens, you should use robocopy, the much better way to copy files with the Windows CLI.

     

    For the third piece, you can definiltey use clusrun to do some tricky deployment kinds of things!  Check out this little batch script (I should update to a PowerShell version now, I guess) I wrote, which uses Clusrun and some Windows Fileshares to copy a file to all nodes in the cluster:

    Code Snippet

    @ECHO off
    SETLOCAL

     

    IF "%2"=="" GOTO error

     

    REM Set local variables
    SET FILE_NAME=%1
    SET DEST_DIR=%2
    SET APP_DIR="C:\TempAppShare"
    SET SHARE_NAME=TempAppShare

     

    REM Make a temporary directory %APP_DIR%
    mkdir %APP_DIR%
    copy /Y %FILE_NAME% %APP_DIR%

    net share %SHARE_NAME%=%APP_DIR% /remark:"Temporary share for distribution of %FILE_NAME%" /grant:%USERDOMAIN%\%USERNAME%,READ

     

    REM create the destination directory on the remote machine
    call clusrun /all if not exist %DEST_DIR% mkdir %DEST_DIR%

     

    REM copy the file to the remote machines
    call clusrun /all copy /Y \\%COMPUTERNAME%\%SHARE_NAME%\%FILE_NAME% %DEST_DIR%\*

     

    REM delete the file share you created

    net share %SHARE_NAME% /delete
    rmdir /s /q %APP_DIR%
    GOTO end

     

    :error
    ECHO Usage: "Deploy Files.bat" FileName Destination

     

    :end
    ENDLOCAL

     

     

     

    Sorry for the detayed response . . . there were some technical problems with the forums!

     

    Friday, February 15, 2008 11:17 PM
    Moderator