Windows HPC Server Deployment, Management, and Administration Forum

Managing user access on a GPGPU HPC Cluster.

Wed, 11 Nov 2009 14:36:18 Z

Hi All,

I have configured a cluster where each node has a decent graphics card and use it for performing parallel rendering. The caveats of using MS HPC in a GPGPU setting is that you must:

1. Login a console session on all nodes with one user login, lets call him gpu-user.

2. gpu-user must also be the user that that submits jobs otherwise the scheduler will fail because it must match the job submitter to the user that created the console session on each node.

This is troublesome as often the data that we want to render on the cluster is sensitive and only certain users have access to it. Thus using a shared user (gpu-user) for job submission/node console login is non-ideal.

What would be more preferable is that part of the job submission would be to login the job submitters user as the console session on each node, and then also lock the console until the job has completed.

Does anyone know if it is possible to achieve this work flow through the job manager and/or with some scripting?

Issue launching Cluster Manager

Thu, 19 Nov 2009 15:43:52 Z

I have a client that cannot launch the HPC Cluster Manager from the head node, or from anywhere else for that manner. It states that the scheduling service cannot be contacted even when running it on the head node. All services are started.

Thanks

Java unattended cluster deployment, UAC Problem

Thu, 19 Nov 2009 09:47:41 Z

Hello,

Some guidelines for unattended Java installation in an Win2008 HPC Cluster environment:

Deactivate UAC on all nodes ; Otherwise the nodes will simple hang, and wait for the UAC acceptance that will never happen. You can omit this by doing the first Java installation by hand via RDesktop login. Afterwards all successive unattended installation will succeed. We have currently no clue why. Perhaps some kind of adaptive UAC?
Best practice is to deactive UAC via a registry key:
%windir%\system32\reg.exe ADD HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System /v EnableLUA /t REG_DWORD /d 1 /f
Reboot the nodes, so that the registry change becomes effective.
All will run smoothly now, if the user installing java was logged in to the nodes at least once. This poses a problem with 20++ cluster nodes. however. The basic point is, that there is no User directory created yet and neither are all temp and AppData paths.
Java kindly ignores any variable defined by the OS, e.g. TEMP or TMP. And gathers its own temp dirs which leads to C:\Users\Username\AppData\LocalLow\Temp and many more.
So the installation fails once more, unless these directories are there.
So you have to create them yourself:
mkdir C:\Users\%USERNAME%\AppData\LocalLow\Temp\
After that the usual JRE unattended deployment should proceed

Have any of you had similar experiences?
I would welcome any comments on that.

Regards,

Johannes

(If I forget to update here, have a look at my blog )

Windows HPC Server 2008 R2 Beta 1 now available

Tue, 17 Nov 2009 00:14:10 Z

Our first Beta release is now available! You can read the full press release at http://www.microsoft.com/presspass/press/2009/nov09/11-16SC09PR.mspx if you're in to reading that kind of thing ;)

Windows HPC Server 2008 R2 delivers productivity, performance and ease-of-use improvements in several areas, including the following:

Improved scalability, with Windows HPC Server 2008 R2 offering out-of-the-box support for deploying, running and managing clusters up to 1,000 nodes
New configuration and deployment options such as diskless boot, mixed-version clusters and support for a remote head node database
Improved system management, diagnostics and reporting including an enhanced heat map, multiple customizable tabs, an extensible diagnostic framework and the ability to create richer custom reports
Improved support for service-oriented architecture (SOA) workloads including a new fire-and-recollect programming model, finalization hooks, improved Java interoperability, automatic restart and failover of broker nodes, and improved management, monitoring, diagnostics and debugging
Message Passing Interface (MPI) and networking enhancements including optimizations for new processors, enhanced support for RDMA over Ethernet and InfiniBand, improved MPI debugging, and a pushbutton HPC LINPACK optimization wizard
New ways to accelerate Microsoft Office Excel workbooks such as support for Cluster-Aware User-Defined Functions and the capability to run distributed Excel 2010 for the cluster

Come and join our beta program to give it a try, and you can give us feedback (positive or negative) on it: http://connect.microsoft.com/HPC/content/content.aspx?ContentID=6923

Pending software updates failure

Wed, 04 Nov 2009 21:15:38 Z

Our compute nodes are on a private, firewalled network. Only our head node and other compute nodes can access the compute nodes. The head node can access anything. When I run the diagnostics, I get the following failure:

Windows update failed to find updates. Exception from HRESULT: 0x80072EFD -2147012867

I followed the instructions to set up a winhttp proxy, and running netsh winhttp show proxy on each compute node gives

Current WinHTTP proxy settings:

Proxy Server(s) : masternode

Bypass List : (none)

masternode is our head node.

Ideally I'd like to "broadcast" Windows Updates (and potentially other software updates) to the compute nodes from the head node.

Any suggestions?

TIA

Compute node stuck in "Provisioning" state

Tue, 13 Oct 2009 19:48:06 Z

Hi,

I am trying to setup a compute node with Windows HPC Server 2008. After changing the network topology, I have a compute node that is stuck in the "Provisioning" state.
I have tried the following to get it unstuck:

* Rebooted both the head node and the compute node.
* Tried to use the Cluster Manager UI to cancel the operations. But the "Cancel" option is greyed out.

The problem is that when a node is in this state, it can't even be deleted (which would allow me to start over!).

Any ideas on how to get out of this?

Thanks,

Brian

Windows HPC 2008 with Infiniband

Wed, 04 Nov 2009 09:54:51 Z

HI, I am new to HPC and need some direction and help on how this would work.

I have been tasked at setting up a server (HP380G5) which we need oracle running, with its database and Logs located on a Storage server (HP180G6) with HP Volitaire Infiniband (24 port switch) linking them up.

I have been getting conflicting reports that this is not possible as infiniband is not compatible to windows, but this site seems to confirm it does work, however will it work in the way we have been tasked to set it up.

If it does help in how to set this up would be very appriciated

Head node as AD Controller with two NICs

Wed, 21 Oct 2009 15:01:54 Z

We have our head node configured as a AD Controller for our clients. One NIC connects to the university's (CCNY) network for client access, and the other connects to the compute nodes on a private network (but still are members of the domain). However, with this setup the DNS server (which must be running for the AD controller) has trouble. It seems that multihomed AD controllers are not generally recommended, but I'm not sure how else to configure this system.

Any tips?

Thanks!

Unable to list HPC users

Wed, 26 Aug 2009 12:00:58 Z

Hello,

Since today some users can connect to the HPC cluster but can't create new jobs. When they try, they get the error message "Access denied".
Some users can work as usual and some jobs are running now.
When we (the administrators) want to display the list of users (Configuration / Users) we see the error message "There was a network problem or the server was disconnected. Please try connecting again". We've tried with different accounts, remotely or locally on the head node. We've restarted the HPC servers. The problem isn't solved.

Diagnostics are ok (All functionnal tests)
Nothing found in the logs.

Any idea?

Thanks for your help.

Marc

PS: SP1 isn't installed.

More on using Unix dhcpd to manage a Windows 2008 HPC Cluster

Mon, 12 Oct 2009 20:12:50 Z

We've decided to replace the Microsoft DHCP server, and just dhcpd to manage both the Windows and Linux sides of our cluster. We've managed to get the Windows nodes to boot by assigning the following configuration to the windows nodes:

  host abe0680 {
     hardware ethernet 00:15:C5:FD:2C:C6;
     fixed-address 10.1.66.172;
     option domain-name-servers 10.1.69.56;
     filename "Boot\x64\WdsNbp.com";
     option vendor-class-identifier "PXEClient";
     next-server 10.1.69.56;
  }

When we want the node to boot into Linux, we apply different (Linux-specific) netboot configuration. This way, we can change the number of nodes that are in Linux or Windows based on our expected workload, with a script and a reboot. This works pretty well.

However, WDS can't image/provision a new node. Does anyone know what the DHCP configuration should look like when we're trying to boot WinPE for provisioning?

Thanks,
-Luke

storage of user's data

Wed, 23 Sep 2009 00:39:13 Z

hi, this is a 2003 cluster. every time a user login to one node, the profile of user was copy to the current node. The problem is that user's "my documents" has data with 30G, and the node has only 20G free space on "C:\Documents and Settings". Therefore, not able to login. How to solve?
Can I use a NAS to storage the user's "my document", thus only one copy of the data. and How to?

is it recommended to install Antivirus on Windows HPC Server 2008

Sun, 25 Oct 2009 12:02:12 Z

is it advisable or recommended to install Symantec Endpoint Antivirus 64bit edition on Windows HPC Server 2008 Headnode.

HPC Pack 2008 SQL Express install error @HPC Server 2008 Eval [Latest Bits available via MS Download]

Thu, 22 Oct 2009 20:16:06 Z

Hi !

I did download the latest available HPC Server 2008 Eval plus the HPC Pack bits and did a clean install of both on to an Intel Core 2 Duo machine with 1.5 GHz and 2 Gig RAM.

Everytime I get stuck during the SQL install part of the setup file with error message:

SQL Server Setup failed to execute a command for server configuration. The error was [Microsoft][SQL Native Client][SQL Server]Cannot add functional unit 'sp_sqlagent_get_startup_info' to component 'Agent XPs'. This unit has been already registered with the component. Refer to the server error logs and Setup logs for detailed error information.

Event Viewer [Application Log]:

Error: 50000 Severity: 20 State: 127 Cannot add stored procedure to component. SYSDBUPG.SQL terminating.

Furthermore I downloaded and tried the SQL Server 2005 standalone, but after install went thru withour any errors this database does not even get recognized by the HPC Pack installer (option to setup new database or use existing one).

Use existing database is greyed out:(

Apart from that I also tried standalone SQL installations. SQL Express 2005 SP2 + SP3 fails too. Even tried some of the tipps already provided here in the forums and from the Internet, but without success.

It is just the SQL part which makes things being stuck, chosing install as non head node and setup of HPC Pack runs thru.

Thanks in advance for all help!

Cheers,

Harry

Question regarding MS HPC 2008 head node failure

Fri, 16 Oct 2009 13:27:45 Z

I am researching MS HPC 2008 as a grid computing solution, and have a question regarding head node operations that I have not yet been able to find an answer to.

In the event of a head node failure (assuming a single head node configuration), what happens to any jobs that are currently in process? Do the jobs fail? If not, do the jobs continue to process until the head node is brought back on-line? Or do jobs that are currently process go into a "suspended" state and resume once the head node is brought back on-line? Is this resumption process automatic, or a manual process?

Basically, I'm trying to make the determination if I need to cluster the head node right away, or if having the head node be a single point-of-failure within the HPC grid is a low-risk decision.

Thanks,
Matt

Scheduling Diagnostics Tests

Mon, 12 Oct 2009 17:27:11 Z

Many of you have asked for this functionality. So we have written and published the doc on how to schedule diagnostic tests using Windows task scheduler: http://technet.microsoft.com/en-us/library/ee649304(WS.10).aspx

Mass activation on cluster - how?

Tue, 11 Aug 2009 16:49:26 Z

For some reason some of the nodes did not activate automatically after deployment. In order to activate them I need to use the slmgr.vbs -skms <servername> and then reboot and then when it asks for activation, I choose - activate online - and then sometimes it works, but still not always.

I have a fairly big cluster and it is difficult going to each node and do that. I am aware of the clusrun command, but I found that slmgr -ato is not working, so I need to actually reboot after -skms and then if I am lucky - it works :)

Any clues?
Regards
Ivan

HPC SP1 Kills DHCP?

Mon, 07 Sep 2009 15:05:18 Z

Hi folks,

After installing SP1 (using windows update -- with all the other updates) on a clean machine with Windows Server 2008 HPC kills DHCP server. It just stops and looks like described here http://support.microsoft.com/kb/895149.

Any ideas?

Thanks,

Andrei

Ganglia for Windows HPC 2008

Tue, 19 May 2009 06:22:07 Z

Hi everyone,

in order to have our Windows Cluster (2003) showing occupation etc. together with our Linux systems, we installed the precompiled ganglia from APR Consulting on the nodes.
However the executables had a severe memory leak on the 2008 OS, so we tried to build our own Windows/Cygwin Ganglia and succeeded.

Its in my interest if anyone else is doing this and what are your experience?

So far we manage to run gmond.exe as a scheduled task once on startup. After that it daemonizes. All other attempts to install it as a service, e.g. cygrunsrv, sc.exe aso. failed.

Of course a full blown Windows Service would be much more appreciated.

For those who want to try and are not so familiar in the Linux/Cygwin environment (I'm not as well) I've posted a small blog article here, which guides through the basic steps:

http://www.blogs.uni-erlangen.de/JohannesHabich/stories/3146/

All links are there, too.

setting CCP_SERVICEREGISTRATION_PATH for SOA service registration

Thu, 01 Oct 2009 16:14:37 Z

Hi,
http://msdn.microsoft.com/en-us/library/cc853430(VS.85).aspx says to use
cluscfg setenvs CCP_SERVICEREGISTRATION_PATH=\\<server>\<share>\<servicename>.config
to register a SOA service.
1. This syntax implies that only one service can be registered, which I find hard to believe. Is the doc incorrect?
2. When I run the command, I get this error: Only Administrator could set cluster environment variables
I am already a member of the Administrators group on the machine. Is there another permission I need to set for my userid?
thanks
Phil

Strange Results with Lizard

Tue, 29 Sep 2009 14:23:34 Z

Im currently running Lizard on a 32 node cluster (16 duel + 16 Quad cors) and at the start 8 of the quads dont run in the verifiaction tests till the rest have finished, this is when only four of them run and it passes the test.

in the conistancy test only the quads run and this passes, normal result is around 90% conistancy and 60GFlops

then in the Tuning process 3 of the dual wont run.

all nodes are online and have fresh reimages, execpt for the head node, which did have the intel cluster tools, but as they seemed to be creating some problems for Lizard they have been taken off for the moment.

the reason that lizard needs to be run is so the i can get results for my MSc project

any help will be greatly received

Deplyoment of Win2008 plain to HyperV R2

Mon, 28 Sep 2009 06:48:49 Z

Hello,
I'm currently rebuilding our entire Windows HPC structure. Therefore I wanted to deploy the Headnodes by WDS, too. So I installed Win2008 R2 Server onto a WDS Virtual Machine hosted by HyperV.
As Hpc20008 Cluster Pack V2 is not compatible with Win2008 R2 I then switched from deploying Win2008 R2 to Win2008 plain.
However the installation never succeded in adding the prestaged node to the Domain:
With a WIn2008 based WDS the deployment goes the same way, no domain join.

The solution seemed to be pretty simple. The boot.wim from Win2008 plain does not have the appropriate drivers for R2's HyperV. With the boot.wim of R2 starting up, the domain join is done and everything runs smoothly.

However the final OS then stops booting with a missing or faulty vmbus.sys.

Injecting VMBUS.sys to any image and injecting all inf drivers of the HyperV client tools disk does not seem to work for the boot.wim or the install.wim either.
Adding the paths to the peboot sequence unattended pass results in a abort of the deployment. But I have to mentioned I have not error checked everything here.

Finally I decided to get this deployed today after all and simple added a secondary generic ethernet device.
Here a small setback was that the VM instantly boots from the newly added adapter and no longer from the first one, this is pretty confusing as each adding of a network adapter means to adjust mac etc. again and again if you want to do PXE boot.

This setup finally worked.

However I would appreciate a way to really inject the necessary drivers for HyperV to boot.wim and install.wim. Perhaps someone knows where to find them or how to do it right?
Her is what I did:

imagex /mountrw boot.wim 2 boot

where boot is a folder in the current directory

peimg /inf="Hyper-V Integration Services\netvsc\netvsc.inf" boot

imagex /unmount /commit boot

Finally the driver itself seems not to be working as hoped, although its the same version as listed as functional and installed inside one HyperV VM.
Tested this way:

drvload.exe \\someshare\netvsc.inf
wpeutil InitializeNetwork

Johannes

Installing PowerShell Version 2.0 on a system with HPC Server 2008 SP1

Tue, 29 Sep 2009 20:01:20 Z

Hello All,

I have a system running Microsoft HPC Server 2008 Service Pack 1. It comes with PowerShell version 1.0. I need functionality available in PowerShell version 2.0. Does anyone know of a way that I can just install PowerShell version 2.0?

Aquil

What happened to my DHCP exclusions?

Mon, 21 Sep 2009 22:04:30 Z

When I configured the HPC Pack 2008, I allowed the GUI to set up a DHCP scope on the headnode. Then I added a number of exclusions using the DHCP admin tool, so as not to stomp over the Linux cluster with which I share a network.

Unfortunately, the exclusions appear to be periodically removed from from the DHCP scope's configuration. Not all of them, mind you -- an exclusion that excludes a single IP address remains, so only the most important exclusions have been removed.

This cannot continue to happen -- when the Windows DHCP server goes rogue this way, it's quite possible to take down the other 1080 nodes of our 1200-node production cluster. We're using about 120 nodes to experiment with Windows, and the rest is a production Linux cluster.

So, my questions are as follows:

Can we remove the windows DHCP server and use the standard Unix dhcpd that we know how to control? In addition to gauranteeing that Windows doesn't go rogue and de-stabilize our Linux cluster, this will make it much easier to make the cluster automatically reboot nodes in to Linux or Windows based on what the users ask for.
How can I look under the hood to find out and control what changes the HPC Pack tools are making to the DHCP configuration? Normally, I can just read the Perl script or whatever that modifies the files, or look at logfiles -- but I haven't found either of those on Windows. Altarnatively, documentation that describes what the HPC Pack does to the Windows DHCP server would allow me to troubleshoot the problem and script around any issues that arise.
The word on the street is that WDS runs a secret DHCP server that sends additional packets on top of the regular DHCP server. Can anyone fill me on how this is supposed to work? Knowing how it's supposed to work may save mme any hours of looking at Wireshark output.

My office-mate reported a variation on this issue on Wed, 09 Apr 2008 18:51:32 -0500, so it's been around a while. Reconfiguring services on a cron-schedule in the background without telling the sysadmin and with no visible way for the admin to control the process is a major problem -- it really is a systems time-bomb. I'd love to go to a totally manual (or scripted)l DHCP configuration (preferably using our existing Unix dhcpd configuration), but I haven't found documentation describing which changes the HPC Pack tools make to the DHCP server, and what methods/interfaces it uses to make those changes.

-Luke

HPC20008 Pack V2 on Windows 2008 R2

Wed, 23 Sep 2009 13:36:39 Z

Hello there,

Is it supported to install the Version 2 HPC2008 Cluster Pack onto Windows 2008 Enterprise Server R2 or will only the next upcoming version V3 alias HPC2008 Pack R2 support this platform?

Regards,

Johannes

How to Integration SharePoint and HPC Server 2008?

Mon, 14 Sep 2009 13:21:27 Z

How to Integration SharePoint and HPC Server 2008?

What happened to HPC SP1 download ?

Fri, 11 Sep 2009 15:32:58 Z

When I go to the following page http://www.microsoft.com/hpc/en/us/default.aspx and select HPC Pack 2008 Service Pack 1 from the top picks I get "We are sorry, the page you requested cannot be found."

"Collecting cluster usage data" failed

Thu, 30 Jul 2009 11:40:43 Z

Hi there,

we're getting EventIDs 6100 in the Windows HPC Server logs during the last weeks, which state "The operation 'Collecting cluster usage data' failed to run correctly. The operation was initiated by the user SYSTEM."

The operations log in the Cluster manager Console and the HPCManagement.log File also contain entrys in this context:

2009/07/30 11:44:15 [9][Error][Change ] Exception:
System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.ComputeCluster.Management.ClusterModel.CollectSchedulerUsage.LogJobInformation(ISession session)
   at Microsoft.ComputeCluster.Management.ClusterModel.CollectSchedulerUsage.Execute(ISession session)
   at Microsoft.SystemDefinitionModel.ChangeAction.Execute(Session session)
2009/07/30 11:44:15 [9][Error][Change ] Object reference not set to an instance of an object.
2009/07/30 11:44:25 [5][Error][Change ] Exception:
System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.ComputeCluster.Management.ClusterModel.CollectSchedulerUsage.LogJobInformation(ISession session)
   at Microsoft.ComputeCluster.Management.ClusterModel.CollectSchedulerUsage.Execute(ISession session)
   at Microsoft.SystemDefinitionModel.ChangeAction.Execute(Session session)
2009/07/30 11:44:25 [5][Error][Change ] Object reference not set to an instance of an object.
2009/07/30 11:44:35 [9][Error][Change ] Exception:
System.NullReferenceException: Object reference not set to an instance of an object.
   at Microsoft.ComputeCluster.Management.ClusterModel.CollectSchedulerUsage.LogJobInformation(ISession session)
   at Microsoft.ComputeCluster.Management.ClusterModel.CollectSchedulerUsage.Execute(ISession session)
   at Microsoft.SystemDefinitionModel.ChangeAction.Execute(Session session)
2009/07/30 11:44:35 [9][Error][Change ] Object reference not set to an instance of an object.
2009/07/30 11:44:35 [9][Error][HpcManagement] Event ChangeExecutionFailed: The operation 'Collecting cluster usage data.' failed to run correctly. The operation was initiated by the user: SYSTEM. The operation can be identified by the GUID: becbfc85-8a05-4bcb-bdf7-7d1bf3c32b71. Using this GUID a log of the operation can be obtained from the hpc powershell command: Get-HpcOperation -id becbfc85-8a05-4bcb-bdf7-7d1bf3c32b71 | Get-HpcOperationLog

Restarting the services or the whole headnode did not help to solve the problem. The usage data in "Charts and reports" seems to be ok.
I cannot find any additional information in the system logs. Any ideas what could be wrong or how to diagnose this problem ?

Best regards,
Michael

HPC Basic Profile web service configuration

Fri, 04 Sep 2009 16:53:15 Z

Has anyone used HPCBasicProfile using x509 certificate authentication?
Are there plans for this in the future - having a ps1 script for setting this up?

Support for Windows Server 2008 R2

Fri, 04 Sep 2009 09:50:21 Z

Hello

I've been experimenting with deployment of Windows HPC Server on Windows Server 2008 R2 (Standard/Enterprise Edition FWIW). A straight deployment seems to work OK, but when attempting an upgrade I've hit some issues. My first question, though:

Is running the HPC pack on Windows Server 2008 R2 a supported configuration?

Thanks

Dan

Question is for Server 2008 Standard edition....Terminal Service Roles & Remote Desktop

Thu, 03 Sep 2009 15:27:15 Z

I install server 2008, also configured Terminal Service roles, enabled remote desktop, configured TS Gateway role, TS Web Access, and file services.
During the configurations, Web Server II was installed because I installed Terminal Server role, but I didn't need the server to be Web Server, it's going to be a file server mainly.
My problem is that I'm not able to connect to the server from either within the same LAN network or from outside the network....the server is under "WORKGROUP".
Can anyone visualize what my problem is?

Can we turn off windown authentication in window 2008 hpc when running a client application to connect a headnode?

Mon, 07 Sep 2009 14:30:49 Z

Hi,

It seems a windown authentication is always required in window 2008 hpc when running a client application to connect a headnode, please see message below. is there any way we can turn off windows authentication or use some security credential without this popup message?

>>Enter the password for 'Domain\useraccount1' to connect to 'headnode':

I tried to add an network account from control panel - manager network user passwords but it still has this message with my window loginID? Any suggestion is highly appreicated!

Have an HA / failover head node? Follow this thread...

Fri, 14 Aug 2009 22:05:29 Z

Hi everyone, I'd like to let you know that we are investigating a problem with the installation of Service Pack 1 on a pair of HA head nodes.

The instructions that we'd posted are incomplete, and don't work in all scenarios. We're working to verify our new set of instructions, and also to enhance the service pack installer to help take care of some special HA-related steps.

When we have the updates available, I'll reply to this thread. Until then you should skip installing the service pack on your HA cluster. (non-HA clusters, and client machines are fine)

Deploy compute node failed when using a custom system image!

Tue, 25 Aug 2009 15:20:36 Z

Hi,

I have followed the steps described in the article "How to Capture a “Master” Compute Node Image Using Node Templates " to deploy a cluster with the customed system image.

When I performed the last step "Deploy new nodes using the master image " , everything seemed ok at the begin following the steps defined in the "Deploy Captured Image.xml " to deploy a new compute node.

The target compute node has booted with the customed system image, and I can login in the system by hand and found the customed image is successfully applied on this node.
(the customed image has included some applicaitons, and all these applications added to the image can run successfully on the new node).

But in the cluster manager on the head node, the provision procedure stopped at the "WindowsSetup " step, and it can not start the next step which is going to add the compute node to the domain.

Has anyone seen this before? Any comments or suggestions would be appreciated!

Thank you very much!

Chu Qiu

How to enable netdirect support in Mellanox IB?

Wed, 26 Aug 2009 17:47:38 Z

When MS field techs were here helping us set up our cluster originally, I remember them running a command across all nodes to enable netdirect support on the Mellanox IB drivers for our Altix xe320 nodes. My notes about it are nowhere to be found, of course. Does anyone know what that command would have been?

Cheers!
-Eric

Add preconfigured node failed during Provisioning

Mon, 24 Aug 2009 13:58:21 Z

Hi!

I have a cluster with 8 compute nodes with Domain (developer.com ) and the head node's name is developer-wccs.developer.com .
In order to backup the domain and DNS server ,I have deployed another windows HPC Server 2008 only with second domain contorller and backup dns server,
which named developer-wccs2.developer.com

Some days before,my origin head node crashed because of disk broken, so I have to recover the cluster. Since developer-wccs2 has the domain and dns , so I depolied the HPC Pack on it and made it as a new head node.

The new head node works ok, and I chosen the topology#1 as the cluster network as same as before.
I created a node template without operation system image and used the node xml to add the existed compute nodes to the new head node, since all the compute nodes need not be deployed again.

When I add a compute node , I got an error when provisioning, and the log is as following:

Time    Message
2009/8/25 2:50:23    Reverted
2009/8/25 2:50:23    Dissasociating template from compute node DEVELOPER\COMPUTE-3-1
2009/8/25 2:50:23    The parent operation is being rolled back
2009/8/25 2:50:23    The parent operation is being rolled back
2009/8/25 2:50:23    The operation failed due to errors during execution.
2009/8/25 2:50:23    The operation failed and will not be retried.
2009/8/25 2:50:23    The compute node failed to execute the operation.
2009/8/25 2:50:23    The Management service encountered an error while performing a change on this node. Access is denied to user 'DEVELOPER\DEVELOPER-WCCS2$'. Check the operation log in the Administration Console for more information.
2009/8/25 2:50:17    Checking the configuration of compute node DEVELOPER\COMPUTE-3-1.
2009/8/25 2:50:17    The compute node failed to execute the operation.
2009/8/25 2:50:17    The Management service encountered an error while performing a change on this node. Access is denied to user 'DEVELOPER\DEVELOPER-WCCS2$'. Check the operation log in the Administration Console for more information.
2009/8/25 2:50:11    Checking the configuration of compute node DEVELOPER\COMPUTE-3-1.
2009/8/25 2:50:11    The compute node failed to execute the operation.
2009/8/25 2:50:11    The Management service encountered an error while performing a change on this node. Access is denied to user 'DEVELOPER\DEVELOPER-WCCS2$'. Check the operation log in the Administration Console for more information.
2009/8/25 2:50:05    Checking the configuration of compute node DEVELOPER\COMPUTE-3-1.
2009/8/25 2:50:05    The compute node failed to execute the operation.
2009/8/25 2:50:05    The Management service encountered an error while performing a change on this node. Access is denied to user 'DEVELOPER\DEVELOPER-WCCS2$'. Check the operation log in the Administration Console for more information.
2009/8/25 2:50:01    Checking the configuration of compute node DEVELOPER\COMPUTE-3-1.
2009/8/25 2:50:01    The compute node failed to execute the operation.
2009/8/25 2:50:01    The Management service encountered an error while performing a change on this node. Access is denied to user 'DEVELOPER\DEVELOPER-WCCS2$'. Check the operation log in the Administration Console for more information.
2009/8/25 2:49:55    Checking the configuration of compute node DEVELOPER\COMPUTE-3-1.
2009/8/25 2:49:55    Associating template Dualboot ComputeNode Template with compute node DEVELOPER\COMPUTE-3-1
2009/8/25 2:49:55    Moving node DEVELOPER\COMPUTE-3-1 from state Unknown to state Provisioning.
2009/8/25 2:49:55    Assigning template Dualboot ComputeNode Template to node COMPUTE-3-1.

I found the core error is "The Management service encountered an error while performing a change on this node. Access is denied to user 'DEVELOPER\DEVELOPER-WCCS2$'. Check the operation log in the Administration Console for more information."

I have tried to turn off the firewall of compute node COMPUTE-3-1, and the same error occupied.

Has anyone seen this before? Any comments or suggestions would be appreciated!

Thank you very much!

Chu Qiu

Can't provision compute nodes in HPC 2008

Tue, 01 Apr 2008 21:10:48 Z

I am using the Windows Server Beta distribution dated 3/4/2008 and the HPC version with the same date. After I have created the template, loaded and chosen the OS image and inserted the Product Key, I boot the compute node, as requested, and watch its boot output. It never receives any DHCP requests and eventually times out saying "Boot Failed" "Press Any Key". Meanwhile, on the head node, the compute node's name appears in the provisioning window with a blank check box as if things were going normally. If I check the box and choose "Provision", the utility registers the comute node with Active Directory, then indicates it is assigning the template to the compute node and just hangs there. Not surprising since the compute node has long since given up listening for any PXE boot invitations on the network. If I reboot the compute node at this point, it still doesn't receive any invitations so eventually times out again and indicates "Boot Failed". The "Getting Started" Guide seems to indicate that there should be an initial PXE boot which then pauses and awaits further communications from the head node, which makes sense, but that is not happening. Let me point out that we also run a Windows 2003 CCS lab on these same platforms and have no problems getting a PXE boot and RIS image installation.

Any assistance would be much appreciated!

Unable to install the ethernet driver during the deployment of compute node

Thu, 13 Aug 2009 13:14:35 Z

Hi!
I'm deploy a cluster with WINDOWS hpc 2008 on the Dell M610 blades.

The blade's ethernet adapter is Broadcom's NetXtreme II 1 Gigabit Ethernet controllers 5709, and I donwload the drivers for the RIS service and windows 2008.
Here's the page:
http://www.broadcom.com/support/ethernet_nic/netxtremeii.php

First time, I give the driver of RIS service in the drivers management and start to deployment.
In winpe step, it successfully load the drivers and get the windows image, and finished to install windows hpc 2008.
But the deployment procedure stopped, and I login to the new node just installed, and I found the ethernet driver is not installed correctly, so it cannot get the ip address and cannot communicaiton with the head node.

I thought that I should provide the driver for the windows 2008, so the second time, I give the both drivers of the RIS and Windows 2008 . But the deployment also failed, the compute node's ethernet driver is not installed correctly too.
When I tried to update the driver with windows drivers, it told me that driver version is same and cannot be updated.

I also make sure the ethernet driver for windows 2008 can be installed correctly.

And I also make sure the ethenet driver for windows 2008 cannot used for WinPE.

Please do me a favor!

Thank you very much!

Chu Qiu

Chu Qiu

WSUS patching and rebooting?

Wed, 12 Aug 2009 18:50:52 Z

I've configured WSUS and the patching/rebooting for my cluster-nodes through the regular GPO channels.

However, there isn't an option that works well for my requirements on the cluster:
2 - Notify for download and notify for install
3 - Auto download and notify for install
4 - Auto download and schedule the install
5 - Allow local admin to choose setting

#2 won't work, since nobody uses the compute node's console. #3 is better, but still won't work since we don't have a user on the console. #4 won't work because this option reboots windows computers in the middle of the night -- this is annoying for an office worker, but a showstopper that makes scientific users very angry (think about a computer being automatically rebooted on day 13 of a 14 day simulation). #5 won't work, since there's no local admin to speak of.

I chose #3 for my nodes, but I have dozens of nodes that don't have all of the patches that they should.

When rebooting manually GUI only gives me the option to "install patches and shutdown". Shutdown /? doesn't say anything about installing patches.

My questions are:

Am I missing any implicit behaviors of Win2k8 or the HPC Pack?
Is there a way to cause the patches to be updated when I reboot a node remotely for some other reason?
Is there a better solution to the patches that require a reboot in this environment?

Thanks,
-Luke

Server failed during deployment, cannot reimage.

Tue, 11 Aug 2009 16:32:53 Z

I have started deployment on 5 nodes when the headnode failed and restarted. When it was up again, the five nodes are still in Provisioning state, but nothing happens. In the Porivisoning log the last entry is "Reverted" and in Properties - "no ongoing operations can be cancelled"; State: Provisioning. In Operations the last entry is "saving node information to file - state, commited". And it has not changed for a long time now.

The nodes them selves were in their waiting loop for authorization from the headnode. I have now swtiched them off and will boot them up again when I can clean them up from the provisioning state so that I can start all over again.

Any clues how to do that?

Regards
Ivan

Win2k8 HPC Server Nodes: How do I disable IPv6 in a node imaging template?

Thu, 06 Aug 2009 15:33:34 Z

How do I disable IPv6 in a deployment services template for Windows HPC Server 2008?

It appears that I can minimally disable IPv6 in the template by adding the following "Run OS command" registry hack:

 %INSTALLDRIVE%\Windows\System32\REG.EXE ADD HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip6\Parameters /v "DisabledComponents" /t REG_DWORD /d 0x000000ff

When I run ipconfig, no IPv6 address are shown, which is good. However, the IPv6 components are still checked in the network connection's properties dialog box

At first glance this appears to be sufficient, but I want to take a belt-and-suspenders approach to disabling IPv6, since the Infiniband Subnet Manager ends up on its knees after a certain number of IPv6 hots are on the network. So I want to make absolutely sure that IPv6 isn't going to be enabled.

Does anyone have suggestions for additional ways to keep IPv6 from coming up during the node-deplyment? Or is the above registry hack really sufficient to make my hosts well-behaved on the Infiniband network -- keeping in mind that the head sysadmin here has a Cat5 flail on his wall. :-)

Thanks,
-Luke