Windows HPC Server Job Submission and Scheduling Forum

Exit code 0

Mon, 23 Nov 2009 12:51:19 Z

Hi there,

we are trying to setup a testing environment with Windows HPC 2008 Cluster three compute-nodes (for now) and a head-node. All nodes have the evaluation licence key.
All nodes are shown in the HPC Manager/Node Management with Online and Health OK state.
As test program we wanted to use the batchpi.exe from the files section, so we made a new job.
Job Details: Name = TEST1, Template = Default, Project = TEST, Priority = Normal, RunOptions = nothing checked,
Resources = Node/Auto/Auto
Task List: My Task, c:\batchpi.exe 1000000000, all folders are set to c:\
Resource Selection: Run this job only on nodes 1, 2, 3, not the head-node
<Submit>
The job starts on the nodes and ends a second later with a failed job message: Task failed during execution with exit code 0. Please check task's output for error details.

In the command line (cmd) it runs without problems. We also tried to run the job with mpiexec -n * c:\batchpi.exe 1000000000 and copied the batchpi.exe to the compute-nodes, but nothing helps :(

What are we doing wrong? Please help.

Regards
Volkie

Job properties not changing when submitted from saved .xml file

Mon, 23 Nov 2009 07:13:41 Z

Hi,

Do the following to reproduce the behaviour:

You create a job and save it to an xml-file.
You create another job from that xml-file.
In job details -> job run options you change the maximum run time.

Bug:
The change is completely ignored when submitting the job.
The change is completely ignored when saving the job to an xml file.
We suppose that there are more bugs like these, and that they have a common source.

Regards,

Johannes

Which jobs query the activation filter

Mon, 23 Nov 2009 11:32:59 Z

Hi,

it was my understanding that the activation filter gets called for every job being in the status queued.
That way I could implement fairshares, priorities and of course licensing issues.
However I just found out with my first own activation filter, that only the job with the highest priority of a user is queried by the activation filter.

Is there a way to change these settings and to issue a call to the activation filter for each and every job which is in the state "scheduled"?

Johannes

NodeGroup not changed if other JobTemplate choosen

Mon, 23 Nov 2009 06:41:28 Z

The following actions show an unexpected and unwanted behaviour, at least in my opinion:

Assign a job-Template (Template0) which requires a NodeGroup. Lets say TestGroup0.
Assign another Template (Template1) which requires another NodeGroup. Lets say TestGroup1.
Create a job with Template0 and save it to an xml-file.
Create a job from this xml-file and change the job template to Template1.

Bug symptoms:
You can't submit the job because of error:
"Job template validation failed: The value of property NodeGroups
is out of range. Update the job an try again."

Bug symptoms continuing:
You have a look at Resource Selection where you will find NodeGroup0
in the selected node groups window.

Bug:
The userinterface is actualised, but the data in the job-files is not.

Work around in this case:
5.: Add another nodegroup to the selected nodes,
and remove it again in the job properties.

6.: The job can be submitted.

In my opinion this is a bug. I want to have different queues implemented by job templates. So e.g. a user tests the job properties with a quick run on my short runtime queue and the resubmits the same job on the long running queue with the intended setup.

Regards,

Johannes

HPC 2008 job suspending/resuming

Sat, 07 Nov 2009 09:23:05 Z

I am used to using SGE on a linux cluster with abaqus, when using this and wanting to add a higher priority job any exisiting jobs need to be suspended by using a specific command so that abaqus frees the flexlm tokens.

Is it possible to do this with HPC or do jobs just have to go until completion, our problem is that some jobs take 2 days to run to completion and to utilize abaqus tokens efficiently we need to be able to suspend and resume them as smaller jobs go into the queue.

Thanks

Paul

how to hidden "Remember this password?" when use job submit command?

Thu, 19 Nov 2009 12:25:31 Z

Hi all,
I meet a problem that when i use "job submit" command, it will show the "remember this password?" which will block the command process, since i use this command remotely by web site.
Is there any method to hidden this, or set a default value for it?

thanks,

x64 program using the IScheduler API in HPC Pack 2008 SDK SP1 got errors

Tue, 27 Oct 2009 10:30:35 Z

Hi:

Below is my program. My platform is Windows Server 2008 Enterprise edition + HPC Pack 2008 SP1 (x64)
For a x86 build of this program, everything works fine!
But for a x64 build, I got the error at pJob->put_Name()
The error is: job->put_Name() failed with 0x80040232.
Can anyone help? Thanks very much!
/////////////////////////////////////////////////////////////////////////////////

#define _WIN32_DCOM

#include <windows.h>
#include <stdio.h>
#include <comutil.h>
#pragma comment(lib, "comsupp.lib")

// The Microsoft.Hpc.Scheduler.tlb and Microsoft.Hpc.Scheduler.Properties.tlb type
// libraries are included in the Microsoft HPC Pack 2008 SDK. The type libraries are
// located in the "Microsoft HPC Pack 2008 SDK\Lib\i386" or \amd64 folder. Include the rename
// attributes to avoid name collisions.
#import <Microsoft.Hpc.Scheduler.tlb> named_guids no_namespace raw_interfaces_only \
rename("SetEnvironmentVariable","SetHpcEnvironmentVariable") \
rename("AddJob", "AddHpcJob")
#import <Microsoft.Hpc.Scheduler.Properties.tlb> named_guids no_namespace raw_interfaces_only

int main(int argc, char **argv)
{
    CoInitializeEx(NULL, COINIT_MULTITHREADED);
    HRESULT hr = S_OK;
    IScheduler *pScheduler = NULL;
    // Get an instance of the Scheduler object.
    hr = CoCreateInstance(__uuidof(Scheduler), // CLSID_Scheduler,
                          NULL,
                          CLSCTX_INPROC_SERVER,
                          __uuidof(IScheduler), // IID_IScheduler,
                          reinterpret_cast<void **>(&pScheduler));
    if(FAILED(hr))
    {
        wprintf(L"CoCreateInstance() failed with 0x%x.\n", hr);
        if(pScheduler)
        {
            pScheduler->Release();
        }
        exit(-1);
    }

    hr = pScheduler->Connect(_bstr_t("localhost"));
    if(FAILED(hr))
    {
        wprintf(L"Connect() failed with 0x%x.\n", hr);
        pScheduler->Release();
        exit(-1);
    }

    ISchedulerJob *pJob = NULL;
    hr = pScheduler->CreateJob(&pJob);
    if(FAILED(hr))
    {
        wprintf(L"CreateJob() failed with 0x%x.\n", hr); fflush(stdout);
        pScheduler->Release();
        exit(-1);
    }

    hr = pJob->put_Name(_bstr_t("MyHPCJob"));
    if(FAILED(hr))
    {
        wprintf(L"job->put_Name() failed with 0x%x.\n", hr); fflush(stdout);
        pJob->Release();
        pScheduler->Release();
        exit(-1);
    }

    ISchedulerTask *pTask = NULL;
    hr = pJob->CreateTask(&pTask);
    if(FAILED(hr))
    {
        wprintf(L"CreateTask() failed with 0x%x.\n", hr); fflush(stdout);
        pJob->Release();
        pScheduler->Release();
        exit(-1);
    }

    hr = pTask->put_Name(_bstr_t("MyHPCTask"));
    if(FAILED(hr))
    {
        wprintf(L"task->put_Name() failed with 0x%x.\n", hr); fflush(stdout);
        pTask->Release();
        pJob->Release();
        pScheduler->Release();
        exit(-1);
    }

    hr = pTask->put_CommandLine(_bstr_t("hostname"));
    if(FAILED(hr))
    {
        wprintf(L"put_CommandLine() failed with 0x%x.\n", hr); fflush(stdout);
        pTask->Release();
        pJob->Release();
        pScheduler->Release();
        exit(-1);
    }

    hr = pJob->AddTask(pTask);
    if(FAILED(hr))
    {
        wprintf(L"AddTask() failed with 0x%x.\n", hr); fflush(stdout);
        pTask->Release();
        pJob->Release();
        pScheduler->Release();
        exit(-1);
    }

    hr = pScheduler->SubmitJob(pJob, _bstr_t(argv[1]), _bstr_t(argv[2]));
    if(FAILED(hr))
    {
        wprintf(L"SubmitJob() failed with 0x%x.\n", hr);
        pTask->Release();
        pJob->Release();
        pScheduler->Release();
        exit(-1);
    }

    pTask->Release();
    pJob->Release();
    pScheduler->Release();

return 0;
}

hpc 2008 supports network driver as workdirectory?

Tue, 03 Nov 2009 05:30:18 Z

Hi all,
Does hpc 2008 support network driver as workdirectory? I mapped a network driver as Z(\\host\sharedname), and set it as workdirectory when submitting job. But the hpc 2008 throwed an exception that this workdiectory cannot be found.

thanks

Running HPC Job Manager/Cluster Manager from outside AD domain

Tue, 27 Oct 2009 18:43:44 Z

I'm trying to use WIN HPC 2008 on a small cluster. My strength is not Win administration, but rather development.
(This was the first time I have setup an AD domain, and I know nearly nothing about it.)

To create the cluster I did have to configure an Active Directory domain, which went fine, but now when I try to run Job or Cluster manager from my development machine, which is configured for a workgroup, not the domain, I get the following error:
"HPC Job Manager: The server has rejected the client credentials."

I suspect this may be related to the fact my dev. box is not part of the domain.

Is there a way to get this to work without forcing my development box to join the clusters domain?

Thank you,
Cameron
PS
Is this the best forum to ask these types of questions?

Using Kerberos Protocol Transition to submit jobs on behalf of users

Wed, 28 Oct 2009 21:23:09 Z

Hi Guys,

We are writing software that interacts with HPC Server and we would like to be able to submit jobs on behalf of users without knowing their password. So far the two options that appear to be viable are to either store a list of the user passwords on the submission node (not particularly secure!), or attempt to use kerberos protocol transition as documented here:

http://technet.microsoft.com/en-us/library/cc739587(WS.10).aspx

Has anyone ever tried this? Any thoughts or suggestions?

Thanks,
Mark

Windows HPC Server 2008 R2 Beta 1 now available

Tue, 17 Nov 2009 00:16:19 Z

Our first Beta release is now available! You can read the full press release at http://www.microsoft.com/presspass/press/2009/nov09/11-16SC09PR.mspx if you're in to reading that kind of thing ;)

Windows HPC Server 2008 R2 delivers productivity, performance and ease-of-use improvements in several areas, including the following:

Improved scalability, with Windows HPC Server 2008 R2 offering out-of-the-box support for deploying, running and managing clusters up to 1,000 nodes
New configuration and deployment options such as diskless boot, mixed-version clusters and support for a remote head node database
Improved system management, diagnostics and reporting including an enhanced heat map, multiple customizable tabs, an extensible diagnostic framework and the ability to create richer custom reports
Improved support for service-oriented architecture (SOA) workloads including a new fire-and-recollect programming model, finalization hooks, improved Java interoperability, automatic restart and failover of broker nodes, and improved management, monitoring, diagnostics and debugging
Message Passing Interface (MPI) and networking enhancements including optimizations for new processors, enhanced support for RDMA over Ethernet and InfiniBand, improved MPI debugging, and a pushbutton HPC LINPACK optimization wizard
New ways to accelerate Microsoft Office Excel workbooks such as support for Cluster-Aware User-Defined Functions and the capability to run distributed Excel 2010 for the cluster

Come and join our beta program to give it a try, and you can give us feedback (positive or negative) on it: http://connect.microsoft.com/HPC/content/content.aspx?ContentID=6923
(posting in the 3 main forums, sorry for the spam if you read all of them :) )

Bug in XML parsing of Job Description files

Tue, 10 Nov 2009 06:02:52 Z

Hi,

I am using Python to write job description files (the .xml files). Overall this is working really well...

But, I am finding that the XML parser that is used in the HPC Job Manager/job scheduler has a problem with the ordering of XML attributes in the Job tag.

Basically, if AutoCalculateMax="true" AutoCalculateMin="true" are put to early in the attribute list, they don't do anything. I found this out because
the Python XML writer, orders the attributes in alphabetical order and these attributes are put first (they start with "A"). Below, I have included an example
that shows this behavior. Just import the XML files into the Job Manager and you will see that it is NOT auto calculating the min/max.

I have heard of XML readers/writers that don't preserve attribute order in round trips (parse, then write), but this is because these tools use something like an unordered data
structure internally for the attributes. But I have never heard of one whose parsing itself depends on the attribute order. Can someone look into this issue?

Cheers,

Brian

<?xml version="1.0" encoding="utf-8"?>
<Job AutoCalculateMax="true" AutoCalculateMin="true" Version="2.000" Name="IPCluster" UnitType="Core" MaxCores="1" MaxNodes="1" MaxSockets="1" MinCores="1" MinNodes="1" MinSockets="1" RunUntilCanceled="false" IsExclusive="false" UserName="GNET\bgranger" JobType="Batch" Priority="Highest" Project="IPython" Owner="GNET\bgranger" xmlns="http://schemas.microsoft.com/HPCS2008/scheduler/">
    <Dependencies/>
    <Tasks>
        <Task CommandLine="\\blue\domainusers$\bgranger\Python\Python25\Scripts\ipcontroller.exe --log-to-file -p default --log-level 10" IsParametric="false" IsRerunnable="true" MaxCores="1" MaxNodes="1" MaxSockets="1" MinCores="1" MinNodes="1" MinSockets="1" StdErrFilePath="controller-err.txt" StdOutFilePath="controller-out.txt" TaskName="Controller" UnitType="Core" WorkDirectory="\\blue\domainusers$\bgranger\.ipython\cluster_default">
            <EnvironmentVariables>
                <Variable>
                    <Name>PYTHONPATH</Name>
                    <Value>\\blue\domainusers$\bgranger\Python\Python25\Lib\site-packages</Value>
                </Variable>
            </EnvironmentVariables>
        </Task>
    </Tasks>
</Job>

Job cancellation and process cleanup

Fri, 30 Oct 2009 22:08:07 Z

I am using Ansoft HFSS v12 on and HPC Server 2008 cluster and have found that if I cancel running job, the running process is killed and is thus unable to clean up after itself. HFSS has an option to do a 'clean stop', which allows the application to complete running tasks, cleanup lock files and other files related to the job, and to flush any pending data to the result set UNC before closing. This is sometimes necessary if, during the run, a solve on a particular frequency or set of frequencies fails to converge on a solution, meaning the rest of the run is wasted as the model needs modification. Without this flushed results data, however, it becomes very difficult to track down the source of the problem

Unfortunately, when HPC Server kills a job, it kills the process(es) involved in a way that prevents this sort of cleanup. I've been told this is just the way it is, but I'm wondering if anyone has seen this issue and done any work to find a way around it.

Thanks in advance!

Jamie

Scheduling DB-bound tasks

Wed, 28 Oct 2009 20:50:24 Z

In addition to per-node and per-core, computationally intensive tasks, our development group would like to use the HPC cluster to schedule jobs that involve significant database usage. While these jobs are not high-performance in the sense of computation, we would still like to schedule and manage these tasks on our cluster. Because the bottleneck is not local, we would like the tasks that compose each job to execute concurrently. Is there a way to schedule a job such that more than one of its tasks execute on the same core?

How are tasks killed when you cancel a job?

Sun, 08 Nov 2009 22:36:49 Z

Hi,

I have a long running process (run as a task inside a job submitted to the scheduler) that installs some signal handlers (I know that windows doesn't have signals like POSIX does, but I am using Python, which abstracts this in a way that make sense). These signal handlers make sure the process cleans up after itself. When I run my process by hand from a regular cmd.exe session, the signal handlers get called when I do Control-C. But, when the process is running on a compute node as a task and I stop the process using "Cancel Job", the task exits without the signal handlers being called. This leaves a huge mess each time the process runs that I have to clean up.

So, questions:

* How exactly does the scheduler stop processes on compute nodes?
* Can this be changed, to basically mimic what a Control-C does (this is like a SIGINT on POSIX).

Cheers,

Brian

Python and Paths: UNC vs mapped drives

Sun, 08 Nov 2009 01:02:45 Z

Hi,

I have read through the previous threads about some of the issues related to UNC paths, Python, etc. This turning out to be a huge problem for us. We are trying to enable an open source Python project (IPython: ipython.scipy.org) to work with the scheduler. In particular we have a command line program that we want to use to submit parallel Python jobs for users (it is actually more complicated than that, but from the scheduler perspective, that is it). I want to make sure I understand the issues related to paths...

Are the following correct descriptions?

* It looks like you have to use a UNC path for the working directory, rather than a mapped drive.

* But, cmd.exe doesn't accept this so it defaults the working dir to C:\Windows.

* But, the environment variables get set correctly (%CCP_WORKDIR%)

* But muchof Python cannot handle UNC directories. I am still trying to figure out what parts can and can't handle UNC paths, but this looks like this might be a huge issue for us.

* It looks like the %PATH% environment variable is completely ignored when finding executables run by tasks. This, in combination with the ban on mapped drives, means that if an executable is on a shared drive, you have to give its UNC path at the command line: \\server\share\program.exe

* Are drives simply never mapped on the compute nodes? From my tests, it looks like sometimes they are. For example, sometimes my python script can do os.chdir(r'z:\\documents') on a compute node. Sometimes not. Am I just asking for trouble here?

* We need a robust UNC based method of finding a users home directory. Currently we use %homedrive%%homepath%, but that gives us the mapped drive version. It looks like %HOMESHARE% might have the information in UNC form, but are we always guaranteed that will give the UNC path to the home directory? What if the home directory for a user is local (not shared) - what is %HOMESHARE% in that case?

Cheers,

Brian

Event Log Error

Wed, 15 Jul 2009 19:16:31 Z

Hi,

when I want to look at the HPC Head Node in the „Microsoft-Windows-HPCServer-Scheduler/Operational“ log of the event view, I get an error
„Some events fields my not display descriptive text because this information cannot be retrieved from the component that raises these events. The component may be misconfigured or corrupted.“
And no Events are shown. Is there a way to fix this issue?

Kind regards,
Sven

hpcbpws.ps1 - Install failed, no suitable client or server authentication certificates discovered

Fri, 18 Sep 2009 19:19:00 Z

I'm trying to commission the HPC Basic Profile Web Service, and running hpcbpws.ps1 returns "Install failed, no suitable client or server authentication certificates discovered". I'd initially followed the steps outlined in "The Windows HPC Server 2008 Cluster in a Linux Environment" and set up the AD Certificate Services role on our head node. Since our institution doesn't allow dynamic DNS, I didn't think I could make it an Enterprise cert server, so I made it a standalone in the wizard, otherwise following all the instructions in the document.

A certificate is shown in the Server Manager (domain-server-CA), but I still got the "no suitable cert" error. I then read the "HPC Basic Profile Web Service Documentation for CTP2" document, and followed the instructions to bind the cert to https in ISS manager, but the error is still the same.

Any ideas? I'll probably remove and add the cert server role as enterprise (if it will let me) next.

Cheers! -Eric

Using Job.SoftwareLicense

Fri, 09 Oct 2009 19:05:26 Z

Hi all,

I'm relatively new to developing interaces with HPC Server 2008. In my current application, I need to utilize the SoftwareLicense property of a job to ensure that only one job is being executed on a node at a given time. There will be other job "types" that will be able to run on that node, just not of the first type. To do this, I want to create a fictitious license for the node and require that the job use that specific license.

When I try implementing this by creating a job with a fake/non-existent license, the job still runs although none of the nodes have that license available. How is this possible? I was expecting that the job would fail due to the fact that none of the available resources/nodes have that license available.

I've scrounged around through the Microsoft documentation, but I can only dig up two sentences on what the SoftwareLicense property does.

Would someone mind clearing this up for me?

Thanks!

-Stefan

Job Resource Usage report pedantry

Thu, 08 Oct 2009 14:42:59 Z

I've been looking through the help and Googling, but I can't find the answer to the following question:
What exactly goes in to the "Total Run Time" column in the Job Resource Usage report?

Total Run Time: The biggest unresolved question is whether this value is the time from when the job was submitted until the time that the job ended? Or is the time from when the job-start? On our other clusters, it's common for jobs to sit around for 24 hours or more before the nodes are available to run it. It would clearly be unfair to charge the user for not using the cluster.

Total CPU Hours: Does this value log just the CPU cycles used, but not the time that the processes were sitting around blocking and waiting for I/O?

Thanks,
-Luke

How to check flexlm license availability before executing a job?

Fri, 25 Sep 2009 02:59:10 Z

Dear all,

In Windows HPC Server 2008 scheduler, is it possible to check for flexlm license availability before each job in the queue is executed?

If it's possible, could someone tell me how??

The program I am running is using flexlm network license and right now the job will run even when the license is not available, so the job will stop right away since license is not available.

I guess there are paid programs out there, but I am looking for free resources.

Thank you.

job template

Thu, 10 Sep 2009 01:36:31 Z

1. Is it possible to create a job template which contains parameter constrains about amount of memory the job will use? ( to prevent server disk trashing/virtual memory swapping in the event that several single core jobs that requires lots of memory run on a single node).

2. When job runs on compute nodes and it needs a temporary directory to write temporary logs/files, how does this gets created? Does it use System's TEMP or User's TEMP directory? Does it gets deleted automatically after the job finished?

thank you for your replies...

Remote Applications

Tue, 15 Sep 2009 14:50:43 Z

Hello All,

I work on an application that has both a client and a server component. When the client starts it request that the server starts. The user can request that the server start locally or that it start remotely. Under Linux, SSH can be used to start remote processes/applications. My question "Is what is the best way to start remote applications under windows?" I know that this question isn't specifically related to Job Submission, but right now I am just searching for ideas.

Regards,

Aquil

Job shrinking / growing with different resource granularity

Tue, 15 Sep 2009 11:23:49 Z

Our cluster is configured to adjust resources automatically to give precedence to jobs with higher priority (graceful pre-emption plus increasing and decreasing of resources).

We have one job (A) with many tasks requiring a single (4-core) socket each. The job is at BelowNormal priority with 1-auto sockets for the job resources.
A second job (B) with normal priority was submitted afterwards. This job has several tasks requiring 1 core, 2 of which queued. The job is set to auto-auto cores .
Neither of the jobs/tasks is set to exclusive resource usage.

We were under the impression that job (A) should shrink whenever it finishes a task if job (B) still has more waiting tasks looking for cores. However, this is seemingly not happening as job (B) has several tasks waiting while job (A) finished several tasks but kept the sockets (all 15 it had) to start new tasks instead of handing the sockets over to (B) which is looking for 2 cores for its two remaining tasks.

Is this the expected behavior or does it look like we have a configuration problem?

On a side note, the activity log of job (A) lists "added 1 core on X" when it should probably be 1 socket.

HPC Version: 2.1.1703.0 on Windows 2008 Server.

PS: adding a third job caused job (A) to give up its socket. Looks like a socket is only given up if all its cores are requested by jobs with higher priority...

Working directory not fully "inherited" by task batch-file commands?

Tue, 08 Sep 2009 18:42:08 Z

Hello all, If I set the working directory to \\server\share, and submit a job with a single task which is a batch file with two commands in it: echo “testing” > out del out It will create the file “out” under \\server\share, but fails on the “del” saying “The file name, directory name, or volume label syntax is incorrect.” If I change the line “del out” to “del \\server\share\out” it succeeds. Whatever I set “Working directory” to, or if I let it default to my profile on the node, it creates “out”, but the “del” never finds it unless I hard-code it. Why must I hard-code the absolute UNC path? Note if I login manually to the node and “cd \\server\share; cmd.exe /c \\server\share\file.bat” it works fine. How can I use relative paths for all files in a task’s batch file? It seems working directory isn’t being used for every command (like del), but only for some (like echo + redirect). This is running 2003 Compute Cluster, Job Manager 1.0.067614. Thanks for any tips.

/stdout with append mode?

Tue, 08 Sep 2009 20:05:09 Z

Hi!
I am running a big number of tests on a windows 2008 R2 HPC Server.

I need to append each output of the tests into one file.

Normally you pipe it like this:
command >> output.txt

i am running my test with:
job submit
and a flag:
/stdout:"out.txt"

but out.txt is overwritten on each new test.
I need to append, how to do it?

Can job files have CommandLine attribute with double quotes?

Wed, 02 Sep 2009 18:48:08 Z

I am generating a job file that has CommandLine attribute that has double quotes. Is there a way to escape these double quotes? Here is a snippet...

<Task Version="2.000"

Name="test1.4528.batch_run"

MinCores="2"

MaxCores="2"

CommandLine="C:\Program Files (x86)\Gizmo\client\bin\gizmo -a %COMPUTERNAME% -s C:\Program Files (x86)\Gizmo -c C:\Program Files (x86)\Gizmo\client\config\reference-config.xml -o gizmo.startscript="main()""

WorkDirectory="C:\Users\aabdullah\gizmo\batch_run\results\200992_141522"

StdOutFilePath="C:\Users\aabdullah\gizmo\batch_run\results\200992_141522\stdout.txt"

StdErrFilePath="C:\Users\aabdullah\gizmo\batch_batch\results\200992_141522\stderr.txt"

UnitType="Core" />

As you can see my gizmo command-line requires an option that specifies a startscript and since there are parenthensis it needs double quotes.

Jobs stuck in configuring state

Tue, 18 Aug 2009 01:37:45 Z

Hi,

I am using microsoft APIs (Microsoft.Hpc.Scheduler ) to submit jobs to HPC job scheduler. Our application is installed on 2 separate clusters. Both clusters are on different networks and different domains. Things are working well on one cluster, but on the other cluster the jobs submitted by our application get stuck in "Configuring" state.

If I right click on the job which is in "Configuring" state (in the HPC cluster manager) and click submit job, it gets submitted but my application keeps waiting for a response.

Any ideas what might be causing this behavior? Is it something related to permissions?

Thanks!
Prashant

How do you determine which mahcines have been allocated for your job?

Mon, 31 Aug 2009 19:15:44 Z

Under systems, like PBS, TORQUE, LSF, etc. When a job is submitted and goes into the "Running" state the Workload Manager sets an environment variable such as PBS_NODEFILE or LSB_MCPU_HOSTS, which contains the names of the nodes that have been allocated for the job. Is there a MS Job Scheduler equivalent? Basically, I've created an application that uses the command-line to create a job and then submit it. I use a jobfile as a "template" and then modify the template based on a users input. The problem is that my application starts a client application that then invokes a call to mpiexec. [NOTE: Here template refers to my application template, note the job scheduler template used to define job policy.] My goal is to make sure that the mpiexec uses the resources allocated for the job. The basic pattern looks as follows:

query user for job details

generate XML job file from a generic xml job file.

create new job:

job new /jobfile:[my_jobfile.xml]

submit job:

job submit /id:[my_job_id]

The CommandLine attribute in the Task element of of my job file starts a an application, which in turn starts an MPI application via mpiexec.

I would like to be able to make sure mpiexec uses the resources allocated to the job.

I know that this is somewhat convoluted, but one of my goals is to create a framework that is generic as possible and allows me ot run under multiple platforms and multiple workload managers. My current abstraction gives me that flexibility so I am hesitant to move to a new implementation unless I can be convinced that it is really the right thing to do.

Thanks for any suggestions.

Aquil

Python Example of using the HPC Basic Profile Web Service

Wed, 26 Aug 2009 23:18:13 Z

Hello All,

I have a python application that I would like to use to drive a parallel application on the Windows HPC Server 2008. I took a look at the C# API provided for communicating with the HPC Basic Profile (HPCBP) Web service, which can run on the head node of a cluster. In the long run, I think that I may choose to go with a .NET implementation, but right now I would like to continue to use python to drive my parallel application.

I know that as a comprimise I could use Iron Python, but right now going that route has it's own set of limitations. So my question is does anyone know of an example, sample, tutorial, or outline for communicating with the HPC Server 2008 Job Scheduler via HPCBP using Python?

Regards

Are custom resources for scheduling possible?

Thu, 06 Aug 2009 19:46:15 Z

I am relatively new to HPC server and I wonder whether it is possible to define custom per-node resources that are used for scheduling (i. e. can be requested by the user) as it is possible with Sun Grid Engine. The background is the following: Our cluster has GPUs for GPGPU, but the number of cores per node is larger than the number of cores. I want to achieve that if all GPUs are in use (but not all cores) no further GPU jobs are assigned to this node (each GPU is an exclusive resource).

Is this possible and how would I do this? If not, are there any plans for such a feature in future releases of HPC server?

And an additional question: Are there any plans to allow initiating console sessions from remote in the future? It would be really cool not to rely on the auto-logon feature and allow per-user sessions.

Best regards,
Christoph

Historical job-queue records?

Wed, 12 Aug 2009 18:37:12 Z

How do I control retention of job records in the Win2k8 HPC scheduling?

We need to keep them around for a long time (a year +) for accounting purposes.

Thanks,
-Luke

failure in "job submit" after HPC Pack SP1 install

Wed, 05 Aug 2009 20:51:22 Z

I can no longer use "job submit" command line and get the following error. Can you tell me what I do to eliminate error?

Database Exception
Procedure or function 'Schd_NextTaskId' expects parameter '@numTasks', which was not supplied.

I need to fix this ASAP so that we can go on.
Our "job submit" was working before the installation of HPC Pack SP1 install.

Cluster Manager Job State stuck on Configuring

Fri, 26 Jun 2009 18:45:03 Z

Hi,
I am testing a SOA style service on HPC 2008 Server sp1 cluster. I have one head node and 2 compute nodes. All nodes have State=Online and Node Health=OK in the Node Management pane of Cluster Manager. I ran diagnostics without any issues.

When my client tries to create a session, the cluster manager's Job Management pane shows 2 items. The first is Job ID 4 WCF service, the second is Job ID 5 WCF service - Broker for service job 4. The State for both is 'Configuring'. There is no furher detail in the Job Details. My client just hangs. I let it run overnight and it never got past the configuring stage.

I need advice on how to troubleshoot. I can't find any more information on what is happening during Configuring state.
thanks
Phil

How to evenly create the processors on the nodes

Mon, 03 Aug 2009 05:50:42 Z

I have a MPI program running on the Windows HPC cluster. The cluster have 50 nodes, each nodes have 8 processors. I tried to run my program using 50 processors. The windows HPC always creates 8 processors on 1 nodes. How can I use 25 nodes and each nodes create 2 processors. I tried to use machine file, however, The windoes HPC didn't create processors as my machine file specified.

How to Submit a job to a HPC cluster via WPF browser Application

Mon, 27 Jul 2009 18:40:46 Z

Hi,

I am using the HPC SDK to submit jobs to the cluster. The app I am trying to use is WPF browser App. One the WPF browser app is published, it works fine on machines which have the HPC Client Tools installed. However I am getting error when the WPF browser app is run from machines which do not have the HPC client tools. I also tired with publishing all the HPC related dlls. Even this does not work. The error I get is as follows:

=========================================
ERROR SUMMARY
Below is a summary of the errors, details of these errors are listed later in the log.
* An exception occurred while determining platform requirements. Following failure messages were detected:
+ Unable to install or run the application. The application requires that assembly Microsoft.Hpc.Scheduler.Store Version 2.0.0.0 be installed in the Global Assembly Cache (GAC) first.
* An exception occurred while downloading the application. Following failure messages were detected:
+ The DeterminePlatformRequirements method failed. The application cannot be committed.

==================================

I know that the alternative would be to use jsdl, but that seems to more cumbersome. I would prefer to use the HPC SDK instead.

Is there a way to get this working without getting the HPC dll's installed in the GAC?

Thanks
Rama

Job is in a "Failed" state but all tasks have a "finished" state

Thu, 09 Jul 2009 06:00:05 Z

We run on HPC 2008 and have a job that indicates that it's failed - and nominates the failed tasks in the error message. However, all tasks for the job have a state of "Finished" (ie there are none that failed). The tasks that the job manager says failed show no sign of failure: they all are Finished; they have no error messages; and they all processed what they were supposed to succesfully.

There's no indication of problems in any event logs and there's no one particular Node that they were running on. (And all other tasks on the nodes were successful)

The job is embarassingly parallel and was running on 3 nodes with 28 cores. There's 271 tasks.

In one instance the job was running for 22 hours. The 5 tasks that failed all started within a 20 minute period. None of the other tasks started during that time. All cores were fully utilised throughout the job so they weren't the only 5 things running at the time.

However, we've also had the problem with the same job (a lot less data) running with the same number of nodes, cores and tasks and running for 5 minutes. And then one of the tasks fails.

We can't reproduce this behaviour at will.

We'd appreciate any thoughts on what's happened and how we can resolve what appears to be incorrect reporting of the state.

Unhandled Exceptions

Tue, 16 Jun 2009 12:30:18 Z

If an application that is being run by the Job Manager throws an unhandled exception, the job will continue to keep running indefinitely. If the application running in a task generates an unhandled exception the just-in-time debugger dialog comes up on the node where the task was running and the job remains in a running state. The only way to get the job to fail is to login to the nodes and close the dialog box, which in turn causes the job to fail. Is there any way to make the jobs fail without interaction with the node if an exception is thrown?

Trace.Assert Performing As Expected When Run From The Cluster Manager

Tue, 16 Jun 2009 18:19:20 Z

Trace.Assert doesn't work as expected when running as a job in cluster manager - Here's a VB.NET code sample that illustrates the problem:

Sub Main(ByVal args() As String)

Console.WriteLine("Before assert")

Trace.Assert(False)

Console.WriteLine("After assert")

End Sub

When logged directly into one of the HPC nodes, if you run the application it prints the first line to the console and then puts up a dialog as expected. But if the application is run as a task in an HPC job, both lines are written to the console and the assert doesn't have any affect. Is this behavior expected?

Task fails with -1073741819

Thu, 09 Jul 2009 08:41:24 Z

I have tasks that fail with code -1073741819 which when translated indicate an access violation but not sure if this is how i interpret the code. Within our HPC environment jobs are executed from a shelled cmd.exe (not sure if this is standard HPC mechanism for running jobs but its certainly ours). So I presume the return code from our program is passed back to cmd.exe which then passes it back to HPC? In any event, I dont know how it picks up this error code because our jobs seem to indicate success.