none
Access Denied - to both Job Files and Output Data RRS feed

  • Question

  • Good Morning,Running Matlab on Windows 2012 Server + HPC Pack a problem has been encountered in which I have been receiving a significant number of instances where file access has been denied whilst attempting to ammed existing files on our SAN.  Looking further into this it would appear the culprit files are those being stored in Uppercase, but have specified the files to be saved as Lowercase equivelant.  Clearly this issue is leading to the simulations crashing and significant data loss.  I attempted to seek advice from the MATLAB froum (they suggested I ask the question here).  The question is how does Windows save files to disk, what is the process by which a file specified in a Lowercase could be stored on the network in Uppercase ?  Any help most appreciated...Regards Paul PS Could other factors be involved here (I know of a number of cases where Windows explorer appear to suggest files exist, which have actually been deleted and vice versa - could the files be written but windows has not updated its file system database hence does not recognise the existance of said files...It all a mess)
    Monday, April 17, 2017 9:54 AM

All replies

  • Hi,

      HPC Pack will only output the file in what you have specified (In lower case or upper case). Could you export an typical job to xml file and share with us?


    Qiufang Shi

    Wednesday, April 19, 2017 1:46 AM
  • Sorry about the delay in replying to you request, the xml file has 252 jobs in total, but for clarity I have removed a large chunk in the middle, but they are all of the same form

    <?xml version="1.0" encoding="utf-8"?>
    <Job Version="3.000" Id="429138" State="Running" SubmitTime="27/04/2017 08:07:57" CreateTime="27/04/2017 08:07:46" StartTime="27/04/2017 08:07:59" Name="MATLAB" IsExclusive="false" RunUntilCanceled="false" UnitType="Core" RuntimeSeconds="604800" Owner="Redacted" UserName="Redacted" Project="Redacted" JobType="Batch" JobTemplate="Redacted" Priority="Normal" NodeGroups="Redacted" SoftwareLicense="" OrderBy="" RequeueCount="0" AutoRequeueCount="0" PendingReason="None" AutoCalculateMax="true" AutoCalculateMin="true" FailOnTaskFailure="false" Progress="26" ProgressMessage="" MinCores="1" MaxCores="1" NotifyOnStart="false" NotifyOnCompletion="false" ParentJobIds="" EstimatedProcessMemory="0" xmlns="http://schemas.microsoft.com/HPCS2008R2/scheduler/">
        <Dependencies />
        <Tasks>
            <Task Version="3.000" Id="40062153" ParentJobId="429138" State="Queued" UnitType="Core" WorkDirectory="C:\" NiceId="1" CommandLine="net use V: \\UKRSCS740\HPC-PROJECTS-V /persistent:no" RequeueCount="0" PendingReason="None" StartValue="1" EndValue="1" IncrementValue="1" GroupId="0" SubmitTime="27/04/2017 08:07:57" StartTime="27/04/2017 08:08:00" CreateTime="27/04/2017 08:07:46" Name="Drive mapping" MinCores="1" MaxCores="1" AutoRequeueCount="0" Type="NodePrep" FailJobOnFailure="false" />
            <Task Version="3.000" Id="40062154" ParentJobId="429138" State="Running" UnitType="Core" NiceId="2" CommandLine="&quot;C:\Program Files\Matlab\R2015a\bin\worker.bat&quot; " StdOutFilePath="\\UKRSCS740\HPC-PROJECTS-V\100B\job-data\Job130\Task1.log" StdErrFilePath="\\UKRSCS740\HPC-PROJECTS-V\100B\job-data\Job130\Task1.log" RequeueCount="1" PendingReason="None" StartValue="0" EndValue="0" IncrementValue="1" GroupId="430432" SubmitTime="27/04/2017 08:07:57" StartTime="27/04/2017 08:19:12" CreateTime="27/04/2017 08:07:46" IsExclusive="false" MinCores="1" MaxCores="1" AutoRequeueCount="1" Type="Basic" FailJobOnFailure="false">
                <EnvironmentVariables>
                    <Variable>
                        <Name>MDCE_STORAGE_CONSTRUCTOR</Name>
                        <Value>makeFileStorageObject</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_STORAGE_LOCATION</Name>
                        <Value>PC%7B%5C%5CUKRSCS740%5CHPC-PROJECTS-V%5C100B%5Cjob-data%7D%3AUNIX%7B%7D%3A</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_JOB_LOCATION</Name>
                        <Value>Job130</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_DECODE_FUNCTION</Name>
                        <Value>parallel.internal.decode.hpcserverSingleTask</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_SYNCHRONOUS_TASK_EVALUATION</Name>
                        <Value>false</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_TASK_LOCATION</Name>
                        <Value>Job130/Task1</Value>
                    </Variable>
                </EnvironmentVariables>
            </Task>
            <Task Version="3.000" Id="40062155" ParentJobId="429138" State="Running" UnitType="Core" NiceId="3" CommandLine="&quot;C:\Program Files\Matlab\R2015a\bin\worker.bat&quot; " StdOutFilePath="\\UKRSCS740\HPC-PROJECTS-V\100B\job-data\Job130\Task2.log" StdErrFilePath="\\UKRSCS740\HPC-PROJECTS-V\100B\job-data\Job130\Task2.log" RequeueCount="1" PendingReason="None" StartValue="0" EndValue="0" IncrementValue="1" GroupId="430432" SubmitTime="27/04/2017 08:07:57" StartTime="27/04/2017 08:20:03" CreateTime="27/04/2017 08:07:46" IsExclusive="false" MinCores="1" MaxCores="1" AutoRequeueCount="1" Type="Basic" FailJobOnFailure="false">
                <EnvironmentVariables>
                    <Variable>
                        <Name>MDCE_STORAGE_CONSTRUCTOR</Name>
                        <Value>makeFileStorageObject</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_STORAGE_LOCATION</Name>
                        <Value>PC%7B%5C%5CUKRSCS740%5CHPC-PROJECTS-V%5C100B%5Cjob-data%7D%3AUNIX%7B%7D%3A</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_JOB_LOCATION</Name>
                        <Value>Job130</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_DECODE_FUNCTION</Name>
                        <Value>parallel.internal.decode.hpcserverSingleTask</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_SYNCHRONOUS_TASK_EVALUATION</Name>
                        <Value>false</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_TASK_LOCATION</Name>
                        <Value>Job130/Task2</Value>
                    </Variable>
                </EnvironmentVariables>
            </Task>
            <Task Version="3.000" Id="40062156" ParentJobId="429138" State="Running" UnitType="Core" NiceId="4" CommandLine="&quot;C:\Program Files\Matlab\R2015a\bin\worker.bat&quot; " StdOutFilePath="\\UKRSCS740\HPC-PROJECTS-V\100B\job-data\Job130\Task3.log" StdErrFilePath="\\UKRSCS740\HPC-PROJECTS-V\100B\job-data\Job130\Task3.log" RequeueCount="1" PendingReason="None" StartValue="0" EndValue="0" IncrementValue="1" GroupId="430432" SubmitTime="27/04/2017 08:07:57" StartTime="27/04/2017 08:20:20" CreateTime="27/04/2017 08:07:46" IsExclusive="false" MinCores="1" MaxCores="1" AutoRequeueCount="1" Type="Basic" FailJobOnFailure="false">
                <EnvironmentVariables>
                    <Variable>
                        <Name>MDCE_STORAGE_CONSTRUCTOR</Name>
                        <Value>makeFileStorageObject</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_STORAGE_LOCATION</Name>
                        <Value>PC%7B%5C%5CUKRSCS740%5CHPC-PROJECTS-V%5C100B%5Cjob-data%7D%3AUNIX%7B%7D%3A</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_JOB_LOCATION</Name>
                        <Value>Job130</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_DECODE_FUNCTION</Name>
                        <Value>parallel.internal.decode.hpcserverSingleTask</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_SYNCHRONOUS_TASK_EVALUATION</Name>
                        <Value>false</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_TASK_LOCATION</Name>
                        <Value>Job130/Task3</Value>
                    </Variable>
                </EnvironmentVariables>
            </Task>
            <Task Version="3.000" Id="40062157" ParentJobId="429138" State="Running" UnitType="Core" NiceId="5" CommandLine="&quot;C:\Program Files\Matlab\R2015a\bin\worker.bat&quot; " StdOutFilePath="\\UKRSCS740\HPC-PROJECTS-V\100B\job-data\Job130\Task4.log" StdErrFilePath="\\UKRSCS740\HPC-PROJECTS-V\100B\job-data\Job130\Task4.log" RequeueCount="1" PendingReason="None" StartValue="0" EndValue="0" IncrementValue="1" GroupId="430432" SubmitTime="27/04/2017 08:07:57" StartTime="27/04/2017 08:21:39" CreateTime="27/04/2017 08:07:46" IsExclusive="false" MinCores="1" MaxCores="1" AutoRequeueCount="1" Type="Basic" FailJobOnFailure="false">
                <EnvironmentVariables>
                    <Variable>
                        <Name>MDCE_STORAGE_CONSTRUCTOR</Name>
                        <Value>makeFileStorageObject</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_STORAGE_LOCATION</Name>
                        <Value>PC%7B%5C%5CUKRSCS740%5CHPC-PROJECTS-V%5C100B%5Cjob-data%7D%3AUNIX%7B%7D%3A</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_JOB_LOCATION</Name>
                        <Value>Job130</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_DECODE_FUNCTION</Name>
                        <Value>parallel.internal.decode.hpcserverSingleTask</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_SYNCHRONOUS_TASK_EVALUATION</Name>
                        <Value>false</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_TASK_LOCATION</Name>
                        <Value>Job130/Task4</Value>
                    </Variable>
                </EnvironmentVariables>
            </Task>
    .
    .
    .
    .
    .
           <Task Version="3.000" Id="40062404" ParentJobId="429138" State="Running" UnitType="Core" NiceId="252" CommandLine="&quot;C:\Program Files\Matlab\R2015a\bin\worker.bat&quot; " StdOutFilePath="\\UKRSCS740\HPC-PROJECTS-V\100B\job-data\Job130\Task251.log" StdErrFilePath="\\UKRSCS740\HPC-PROJECTS-V\100B\job-data\Job130\Task251.log" RequeueCount="0" PendingReason="None" StartValue="0" EndValue="0" IncrementValue="1" GroupId="430432" SubmitTime="27/04/2017 08:07:57" StartTime="27/04/2017 08:35:13" CreateTime="27/04/2017 08:07:46" IsExclusive="false" MinCores="1" MaxCores="1" AutoRequeueCount="0" Type="Basic" FailJobOnFailure="false">
                <EnvironmentVariables>
                    <Variable>
                        <Name>MDCE_STORAGE_CONSTRUCTOR</Name>
                        <Value>makeFileStorageObject</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_STORAGE_LOCATION</Name>
                        <Value>PC%7B%5C%5CUKRSCS740%5CHPC-PROJECTS-V%5C100B%5Cjob-data%7D%3AUNIX%7B%7D%3A</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_JOB_LOCATION</Name>
                        <Value>Job130</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_DECODE_FUNCTION</Name>
                        <Value>parallel.internal.decode.hpcserverSingleTask</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_SYNCHRONOUS_TASK_EVALUATION</Name>
                        <Value>false</Value>
                    </Variable>
                    <Variable>
                        <Name>MDCE_TASK_LOCATION</Name>
                        <Value>Job130/Task251</Value>
                    </Variable>
                </EnvironmentVariables>
            </Task>
        </Tasks>
    </Job>


    Thursday, April 27, 2017 5:29 PM
  • I read your first post carefully. The looks like some incompatibility between your SAN file system with the windows tools. For example this post mentioned a similar issue. I think you have couple options:

    1. Setting your SAN file system into "case insensitive" so that it won't be puzzled when client tries access lowercase file while specifying uppercase file name. if you're using SAMBA configuration, just change the configure through smb.conf

    2. If you know what files to access in the job, have a task running before all other task starts, the task is doing file check and rename the file in appropriate case

    3. Disable the "PrettyPath", not sure if this works in your shared SAN, please check: http://www.techsupportalert.com/content/how-force-your-windows-file-and-folder-names-have-case-you-want.htm

    And you still have issues, you may check the windows file system forum.


    Qiufang Shi

    Friday, April 28, 2017 10:26 AM